Oct 30 00:40:08.881298 ip-10-0-143-7 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 00:40:08.881313 ip-10-0-143-7 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 00:40:08.881323 ip-10-0-143-7 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 00:40:08.881704 ip-10-0-143-7 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 00:40:19.078438 ip-10-0-143-7 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 00:40:19.078457 ip-10-0-143-7 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8e057b7530514f1a8d8c4c520ba04a55 -- Oct 30 00:42:18.127137 ip-10-0-143-7 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:42:18.537230 ip-10-0-143-7 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:18.537230 ip-10-0-143-7 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:42:18.537230 ip-10-0-143-7 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:18.537230 ip-10-0-143-7 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:18.540503 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540372 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.540503 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540504 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540508 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540511 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540514 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540517 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540520 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540522 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540525 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540527 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540530 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540533 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540535 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540538 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540540 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540543 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540546 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540548 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540552 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540554 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540557 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.540569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540559 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540563 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540568 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540573 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540576 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540579 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540586 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540589 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540592 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540594 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540597 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540599 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540602 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540604 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540607 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540609 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540612 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540614 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540617 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.541044 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540619 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540622 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540624 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540627 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540629 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540632 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540635 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540637 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540640 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540642 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540645 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540655 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540659 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540661 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540664 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540668 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540671 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540674 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540676 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.541519 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540679 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540681 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540684 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540686 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540689 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540692 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540695 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540697 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540700 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540702 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540704 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540707 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540710 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540712 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540715 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540717 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540720 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540722 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540725 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.541977 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540727 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540730 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540732 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540735 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540737 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540740 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540745 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540748 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540751 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540754 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540757 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540760 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540762 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540765 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540767 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540773 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540775 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540778 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540780 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540782 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.542424 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540785 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540787 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540790 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540794 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540796 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540799 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540802 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.542908 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.540804 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543049 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543056 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543059 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543062 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543065 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543067 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543071 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543073 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.543072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543076 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543079 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543081 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543085 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543087 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543090 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543093 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543097 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543101 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543104 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543107 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543110 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543112 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543115 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543117 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543120 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543122 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543125 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543128 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.543291 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543130 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543133 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543135 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543138 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543140 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543142 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543146 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543149 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543152 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543155 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543157 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543160 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543163 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543165 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543168 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543170 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543173 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543176 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543179 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.543742 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543181 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543184 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543186 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543189 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543193 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543196 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543198 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543201 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543203 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543206 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543208 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543211 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543213 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543216 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543218 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543221 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543223 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543226 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543228 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543231 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.544218 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543234 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543237 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543239 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543242 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543245 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543247 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543250 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543253 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543256 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543258 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543261 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543264 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543266 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543269 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543272 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543275 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543277 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543280 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543282 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543284 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.544685 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543287 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543289 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543292 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543295 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543297 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543300 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543303 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543306 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543308 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543311 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543313 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543317 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543320 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543322 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543325 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543332 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543336 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543339 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543341 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.545190 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543344 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543837 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543844 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543847 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543852 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543869 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543872 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543875 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543878 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543882 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543884 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543887 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543889 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543892 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543895 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543897 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543900 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543903 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543905 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543908 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.545649 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543910 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543913 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543916 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543918 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543921 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543924 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543926 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543929 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543932 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543935 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543937 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543940 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543943 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543946 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543949 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543951 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543954 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543957 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543959 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.546133 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543962 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543965 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543967 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543970 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543974 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543977 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543980 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543983 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543987 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543990 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543993 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.543997 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544000 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544003 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544006 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544009 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544011 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544014 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544017 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.546581 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544020 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544022 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544025 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544027 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544030 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544033 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544036 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544039 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544042 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544044 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544047 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544050 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544053 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544056 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544058 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544060 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544063 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544066 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544068 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544071 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.547072 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544073 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544076 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544078 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544081 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544084 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544086 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544089 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544091 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544094 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544097 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544100 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544102 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544104 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544107 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544109 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544112 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544114 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544117 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544119 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544122 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.547535 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544124 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544127 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544129 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544132 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544134 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544144 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544146 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544149 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.544151 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545102 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545111 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545117 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545122 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545126 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545130 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545134 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545139 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545142 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545144 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545148 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545151 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545154 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:42:18.548013 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545157 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545160 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545163 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545166 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545168 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545173 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545176 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545179 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545182 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545186 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545189 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545192 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545196 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545199 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545202 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545205 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545208 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545211 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545214 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545219 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545221 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545224 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545227 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545230 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545233 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:42:18.548545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545237 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545240 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545243 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545246 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545250 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545254 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545257 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545261 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545264 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545267 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545269 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545272 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545275 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545278 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545281 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545283 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545287 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545291 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545294 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545297 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545300 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545303 2572 flags.go:64] FLAG: --help="false" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545306 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545309 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:42:18.549160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545313 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545316 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545319 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545322 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545325 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545328 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545331 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545334 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545337 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545340 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545343 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545346 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545349 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545352 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545355 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545358 2572 flags.go:64] FLAG: --lock-file="" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545361 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545364 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545367 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545372 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545375 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545378 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545381 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 00:42:18.549729 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545384 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545387 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545390 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545393 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545397 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545400 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545404 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545408 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545411 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545414 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545417 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545420 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545426 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545429 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545436 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545439 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545442 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545445 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545448 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545451 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545454 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545457 2572 flags.go:64] FLAG: --port="10250" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545460 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545463 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-009549338b3f1a679" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545466 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545469 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:42:18.550388 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545472 2572 flags.go:64] FLAG: --register-node="true" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545475 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545479 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545482 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545485 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545488 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545491 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545494 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545497 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545500 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545503 2572 flags.go:64] FLAG: --runonce="false" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545506 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545509 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545512 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545515 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545518 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545521 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545524 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545528 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545531 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545534 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545537 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545540 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545544 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545547 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:42:18.551026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545550 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545556 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545559 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545561 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545566 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545568 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545571 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545574 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545577 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545580 2572 flags.go:64] FLAG: --v="2" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545584 2572 flags.go:64] FLAG: --version="false" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545588 2572 flags.go:64] FLAG: --vmodule="" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545592 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.545595 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545728 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545732 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545735 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545738 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545741 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545744 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545747 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545750 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545752 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.551654 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545755 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545758 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545762 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545765 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545767 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545770 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545772 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545776 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545779 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545782 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545785 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545787 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545790 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545792 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545795 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545798 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545800 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545803 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545805 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.552234 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545808 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545811 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545813 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545816 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545818 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545821 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545824 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545826 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545828 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545831 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545834 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545836 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545839 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545841 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545844 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545847 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545851 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545853 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545870 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545873 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.552682 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545878 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545880 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545883 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545885 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545888 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545891 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545893 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545896 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545898 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545901 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545903 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545906 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545909 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545911 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545914 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545917 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545919 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545922 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545924 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.553235 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545927 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545929 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545932 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545934 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545936 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545939 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545942 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545944 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545948 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545950 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545954 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545956 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545960 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545963 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545966 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545968 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545971 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545974 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545976 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.553973 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545980 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545984 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545987 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545990 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545992 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545995 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.545997 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546000 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546002 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546005 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546007 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546011 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546015 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546018 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546021 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546023 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546026 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546029 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546031 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.554504 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546034 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.546060 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546097 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546103 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546106 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546109 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546113 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546118 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546122 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546125 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546128 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546131 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546134 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546136 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546139 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546142 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.554983 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546145 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546149 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546152 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546156 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546159 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546161 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546164 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546166 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546169 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546171 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546174 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546176 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546179 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546181 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546184 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546186 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546189 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546192 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546195 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.555360 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546198 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546200 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546203 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546206 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546209 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546212 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546214 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546217 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546219 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546222 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546224 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546227 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546229 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546231 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546234 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546236 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546239 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546242 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546244 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.555802 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546247 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546249 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546252 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546254 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546257 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546259 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546261 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546264 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546266 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546269 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546271 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546274 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546276 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546279 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546281 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546284 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546286 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546291 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546293 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546296 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.556267 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546298 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546301 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546303 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546306 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546308 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546310 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546313 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546315 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546318 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546320 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546323 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546325 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546327 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546330 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546332 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546335 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546337 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546340 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546342 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.556730 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546345 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546347 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546350 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546352 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546354 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546357 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546360 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546363 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546365 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546367 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546371 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546374 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546376 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546379 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.546381 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.553721 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.553745 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.554366 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.554387 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554652 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.557197 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554662 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554667 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554670 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554673 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554677 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554680 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554683 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554686 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554688 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554691 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554693 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554696 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554699 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554702 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554707 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554710 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554712 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554715 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554717 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.557687 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554720 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554722 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554725 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554727 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554730 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554732 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554735 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554738 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554740 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554742 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554745 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554749 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554752 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554754 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554757 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554759 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554761 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554764 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554768 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554773 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.558170 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554776 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554779 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554783 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554786 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554789 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554793 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554796 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554798 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554801 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554804 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554806 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554809 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554812 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554814 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554817 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554819 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554822 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554825 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554827 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.558773 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554830 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554834 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554837 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554840 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554843 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554845 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554848 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554850 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554853 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554868 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554871 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554874 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554876 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554879 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554882 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554884 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554887 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554889 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554892 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.559236 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554895 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554897 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554900 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554902 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554904 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554907 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554909 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554912 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554914 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554916 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554919 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554922 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554924 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554927 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554930 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554932 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554935 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554937 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554940 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554942 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.559675 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554944 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554947 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554950 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554952 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554954 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554957 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.554959 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.554987 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555147 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555153 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555156 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555159 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555162 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555165 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555167 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555170 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555173 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.560157 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555175 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555178 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555180 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555183 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555185 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555188 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555191 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555193 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555196 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555198 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555201 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555204 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555206 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555209 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555211 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555214 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555216 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555218 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555221 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555223 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.560569 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555226 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555228 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555231 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555233 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555236 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555238 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555242 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555246 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555249 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555259 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555262 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555265 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555267 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555270 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555272 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555274 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555277 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555280 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555282 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555285 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.561049 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555287 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555290 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555292 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555295 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555297 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555300 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555303 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555306 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555309 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555312 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555314 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555317 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555319 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555322 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555324 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555329 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555331 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555334 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.561520 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555337 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555339 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555342 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555344 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555352 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555355 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555358 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555360 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555363 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555366 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555368 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555371 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555374 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555376 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555379 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555381 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555384 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555386 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555389 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.562053 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555391 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555394 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555396 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555398 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555401 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555403 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555406 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555408 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555411 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555413 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555417 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555420 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555422 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555425 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555427 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555430 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555432 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555435 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555443 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.562502 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555445 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.555468 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555508 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555512 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555515 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555518 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555521 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555523 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555526 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555528 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555531 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555533 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555535 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555538 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555540 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555543 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:18.562957 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555545 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555548 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555551 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555553 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555555 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555558 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555561 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555565 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555568 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555570 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555573 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555575 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555578 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555580 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555583 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555585 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555595 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555599 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555603 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:18.563324 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555606 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555608 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555611 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555613 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555616 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555618 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555621 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555623 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555626 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555628 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555631 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555634 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555636 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555639 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555642 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555644 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555647 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555651 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555654 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555657 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:18.563762 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555661 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555664 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555666 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555669 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555671 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555673 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555676 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555678 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555681 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555683 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555691 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555694 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555696 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555699 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555701 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555704 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555707 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555709 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555711 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:18.564263 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555714 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555716 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555719 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555721 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555724 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555726 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555729 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555732 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555734 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555736 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555739 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555741 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555744 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555747 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555750 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555752 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555755 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555757 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555760 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:18.564696 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555762 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555765 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555767 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555770 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555772 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555780 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555783 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555785 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555788 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555790 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555794 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555796 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555799 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555801 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:18.555804 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.555873 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.559708 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.560661 2572 server.go:1014] "Starting client certificate rotation" Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.560761 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:42:18.565144 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.560806 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:42:18.565623 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.562677 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:42:18.565623 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.563898 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:42:18.571753 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.571737 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:42:18.574895 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.574879 2572 log.go:25] "Validated CRI v1 image API" Oct 30 00:42:18.575797 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.575775 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:42:18.579238 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.579208 2572 fs.go:135] Filesystem UUIDs: map[538e9656-c3e4-4fef-bc50-c7ef99eacda6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ed33f9d1-7352-4cee-bb52-8c6d18887643:/dev/nvme0n1p3] Oct 30 00:42:18.579298 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.579239 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:42:18.584602 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.584488 2572 manager.go:217] Machine: {Timestamp:2026-10-30 00:42:18.583274562 +0000 UTC m=+0.338094464 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096261 MemoryCapacity:32811999232 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b693d8cdec017eec19fae2806f928 SystemUUID:ec2b693d-8cde-c017-eec1-9fae2806f928 BootID:8e057b75-3051-4f1a-8d8c-4c520ba04a55 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:7d:3e:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:7d:3e:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:cb:61:76:93:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811999232 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:42:18.584602 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.584603 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:42:18.584706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.584689 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:42:18.585712 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.585686 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:42:18.585888 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.585717 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:42:18.585938 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.585892 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:42:18.585938 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.585902 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:42:18.585938 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.585916 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:42:18.586672 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.586660 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:42:18.587816 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.587806 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:42:18.590561 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.590550 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:42:18.590695 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.590686 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:42:18.591372 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.591362 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:42:18.591425 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.591374 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:42:18.591425 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.591407 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:42:18.591515 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.591468 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:42:18.591515 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.591479 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:42:18.592151 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.592137 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:42:18.594943 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.594914 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:42:18.599684 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.599668 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:42:18.600459 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600445 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:42:18.600508 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600477 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:42:18.600658 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600647 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:42:18.600699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600666 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:42:18.600699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600672 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:42:18.600699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600678 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:42:18.600699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600683 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:42:18.600699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.600689 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:42:18.601344 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601334 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:42:18.601381 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601345 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:42:18.601381 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601353 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:42:18.601381 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601359 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:42:18.601381 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601368 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:42:18.601381 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601377 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:42:18.601985 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601975 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:42:18.601985 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.601985 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:42:18.606035 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.606022 2572 server.go:1263] "Started kubelet" Oct 30 00:42:18.606158 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.606127 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:42:18.606226 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.606148 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:42:18.606226 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.606221 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:42:18.607058 ip-10-0-143-7 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:42:18.608327 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.608314 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:42:18.608559 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.608540 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:42:18.611396 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.611377 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:42:18.612634 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.612615 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:42:18.613099 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613075 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:42:18.613278 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.613248 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-7.ec2.internal\" not found" Oct 30 00:42:18.613758 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613739 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:42:18.613758 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613760 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:42:18.613912 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613741 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:42:18.613912 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613876 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:42:18.613912 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613885 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:42:18.614056 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613936 2572 factory.go:55] Registering systemd factory Oct 30 00:42:18.614056 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.613948 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:42:18.614308 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.614297 2572 factory.go:153] Registering CRI-O factory Oct 30 00:42:18.614308 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.614308 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 00:42:18.614387 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.614351 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:42:18.614387 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.614364 2572 factory.go:103] Registering Raw factory Oct 30 00:42:18.614387 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.614374 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 00:42:18.615366 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.615347 2572 manager.go:319] Starting recovery of all containers Oct 30 00:42:18.626336 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.626316 2572 manager.go:324] Recovery completed Oct 30 00:42:18.631134 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.631117 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.632958 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.632939 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.633050 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.632973 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.633050 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.632989 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.633524 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.633507 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.633603 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.633518 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:42:18.633603 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.633534 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:42:18.633603 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.633550 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:42:18.634239 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634224 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.634308 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634249 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.634308 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634258 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.634613 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634602 2572 policy_none.go:50] "Start" Oct 30 00:42:18.634645 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634617 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:42:18.634645 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.634627 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:42:18.635782 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.635766 2572 policy_none.go:44] "Start" Oct 30 00:42:18.636107 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.636078 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-7.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:42:18.636107 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.636081 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-7.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:42:18.637100 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.636076 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-7.ec2.internal.18e327e469e1443d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-7.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-7.ec2.internal,},FirstTimestamp:2026-10-30 00:42:18.605986877 +0000 UTC m=+0.360806778,LastTimestamp:2026-10-30 00:42:18.605986877 +0000 UTC m=+0.360806778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-7.ec2.internal,}" Oct 30 00:42:18.652852 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.652769 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-7.ec2.internal.18e327e46b7ccfbd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-7.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-7.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-7.ec2.internal,},FirstTimestamp:2026-10-30 00:42:18.632957885 +0000 UTC m=+0.387777787,LastTimestamp:2026-10-30 00:42:18.632957885 +0000 UTC m=+0.387777787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-7.ec2.internal,}" Oct 30 00:42:18.653012 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.652994 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:42:18.653062 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.653006 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:42:18.653062 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.653044 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wrwd" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.681154 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.685575 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688444 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.688480 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688490 2572 server.go:87] "Starting device plugin registration server" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688779 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688792 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688904 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688977 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.688985 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.689593 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.689647 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-7.ec2.internal\" not found" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.691413 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.692905 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.692924 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.692940 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:42:18.693248 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.692969 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:42:18.704275 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.704258 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:18.706692 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.706678 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:18.789085 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.789024 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.793518 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793499 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal"] Oct 30 00:42:18.793608 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793555 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.793801 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793779 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.793919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793813 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.793919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793828 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.793919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.793854 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.794397 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.794374 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.794457 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.794410 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.794457 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.794425 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.796029 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.796017 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.796167 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.796153 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.796211 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.796181 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.797160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797144 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.797242 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797171 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.797242 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797185 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.797242 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797204 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.797242 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797228 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.797242 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.797242 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.798258 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.798242 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.798305 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.798272 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:18.798977 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.798955 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:18.799069 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.798981 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:18.799069 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.798991 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:18.810547 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.810532 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.810635 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.810550 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-7.ec2.internal\": node \"ip-10-0-143-7.ec2.internal\" not found" Oct 30 00:42:18.813309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.813294 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.814981 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.814962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.815046 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.814994 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.815046 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.815027 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.815193 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.815181 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.818173 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.818155 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.839532 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.839512 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:18.840267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.840232 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:18.840423 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.840408 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.843778 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.843760 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:18.843883 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.843801 2572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.868254 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.868240 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:18.868339 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:18.868269 2572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.915819 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.915916 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915826 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.915916 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.915916 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915911 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.916011 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:18.916011 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:18.915922 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:19.145928 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.145901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:42:19.148031 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.148012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:42:19.560954 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.560918 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:42:19.561760 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.561115 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:42:19.561760 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.561122 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:42:19.591833 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.591801 2572 apiserver.go:52] "Watching apiserver" Oct 30 00:42:19.613529 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.613496 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:42:19.632403 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.632360 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767c439e26e5517d9df3fdb3ccab0ad7.slice/crio-7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad WatchSource:0}: Error finding container 7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad: Status 404 returned error can't find the container with id 7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad Oct 30 00:42:19.632910 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.632889 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bade25a7c8ebe0a0dcc7613db38300.slice/crio-78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f WatchSource:0}: Error finding container 78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f: Status 404 returned error can't find the container with id 78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f Oct 30 00:42:19.638394 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.638380 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:42:19.651222 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.651196 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:19.651334 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.651298 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:42:19.651701 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.651682 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6qb5x","openshift-image-registry/node-ca-kp9nh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal","openshift-multus/multus-additional-cni-plugins-k98jz","openshift-network-diagnostics/network-check-target-x9kzn","openshift-network-operator/iptables-alerter-mst6g","openshift-ovn-kubernetes/ovnkube-node-tcd47","openshift-multus/network-metrics-daemon-mrgqt","openshift-multus/multus-g7tv9","kube-system/global-pull-secret-syncer-ncxvc","kube-system/konnectivity-agent-ts2gs","kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2gkzk"] Oct 30 00:42:19.653753 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.653739 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.654874 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.654842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.655944 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.655929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.657334 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.657320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.657943 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.657916 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:37:19 +0000 UTC" deadline="2028-06-10 12:10:14.503266966 +0000 UTC" Oct 30 00:42:19.657994 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.657943 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14147h27m54.845326649s" Oct 30 00:42:19.658229 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.658212 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:19.658300 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.658263 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:19.659224 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.659208 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.660462 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.660398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.661845 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.661827 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.661971 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.661943 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:19.663154 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.663136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.664315 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.664300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.664392 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.664374 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:19.665458 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.665442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.667324 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.667304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.669578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.669557 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.669578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.669558 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.669700 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.669610 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\"" Oct 30 00:42:19.670199 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.670174 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:42:19.670312 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.670241 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.670586 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.670544 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\"" Oct 30 00:42:19.670586 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.670586 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.671345 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.671327 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:42:19.671419 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.671332 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.671419 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.671348 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.671419 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.671387 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\"" Oct 30 00:42:19.674259 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.674245 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:42:19.696796 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.696752 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerStarted","Data":"7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad"} Oct 30 00:42:19.698130 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.698108 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" event={"ID":"09bade25a7c8ebe0a0dcc7613db38300","Type":"ContainerStarted","Data":"78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f"} Oct 30 00:42:19.715275 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.715260 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:42:19.720889 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.720853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.720971 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.720904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.720971 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.720931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.720971 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.720957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721104 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.720980 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721104 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5mr\" (UniqueName: \"kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721104 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.721104 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.721267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721126 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.721267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4w7\" (UniqueName: \"kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.721267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.721267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.721267 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmnj\" (UniqueName: \"kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721315 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721334 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lbv\" (UniqueName: \"kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721374 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721548 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwl4\" (UniqueName: \"kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721630 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llh49\" (UniqueName: \"kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721739 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721768 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.721798 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njcm\" (UniqueName: \"kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721974 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.721993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722017 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722073 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.722364 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722153 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722290 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722326 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722413 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722442 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722676 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fhd\" (UniqueName: \"kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723125 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722798 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.722977 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.723011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.723038 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.723063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.723100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.723546 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.723125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khz2k\" (UniqueName: \"kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.762885 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.762853 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:42:19.762983 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.762952 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fdsm" Oct 30 00:42:19.771467 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771352 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:42:19.771467 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771409 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.771467 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771424 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:42:19.771467 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771466 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771512 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771517 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771555 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771561 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771561 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771565 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771597 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771610 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.771793 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.771784 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:42:19.776905 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.776891 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\"" Oct 30 00:42:19.777009 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.776996 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.777009 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.776997 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\"" Oct 30 00:42:19.783809 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.783792 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\"" Oct 30 00:42:19.783892 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.783795 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:42:19.784088 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.783967 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:42:19.784243 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.784223 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:42:19.786643 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.786623 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:42:19.786752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.786624 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\"" Oct 30 00:42:19.786752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.786632 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:42:19.786752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.786708 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:42:19.823403 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.823403 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.823403 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823447 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823459 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823493 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.823578 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823526 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fhd\" (UniqueName: \"kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823724 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823729 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823766 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823808 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823840 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.823929 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823965 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.823983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824079 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824166 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824186 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824238 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824255 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824371 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824380 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.824395 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824441 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824452 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824504 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824577 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824630 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz2k\" (UniqueName: \"kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824698 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824718 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824760 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824788 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5mr\" (UniqueName: \"kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.824957 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824799 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824803 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824884 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.824915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.824931 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.825009 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:20.324988739 +0000 UTC m=+2.079808645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4w7\" (UniqueName: \"kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmnj\" (UniqueName: \"kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825147 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.825841 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825272 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lbv\" (UniqueName: \"kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825440 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825441 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825498 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825575 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwl4\" (UniqueName: \"kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825665 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.826674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825681 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.825569 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825730 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825738 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825726 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825790 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825801 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825770 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825834 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.825891 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:20.325854554 +0000 UTC m=+2.080674458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825908 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llh49\" (UniqueName: \"kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825963 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.825985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826006 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.827590 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826092 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826345 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njcm\" (UniqueName: \"kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826509 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826516 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.829706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826603 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826664 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826698 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826761 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826977 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.826990 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.830516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827046 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827287 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827344 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827437 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827529 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827629 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827668 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827716 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827722 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827744 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.827758 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.828005 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.828188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.828393 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:19.831349 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.829406 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.843830 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.843799 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:19.843830 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.843818 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:19.843830 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.843828 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:19.843980 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:19.843914 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:20.343894027 +0000 UTC m=+2.098713935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:19.846675 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.846639 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fhd\" (UniqueName: \"kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:19.846849 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.846828 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz2k\" (UniqueName: \"kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.847357 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.847337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5mr\" (UniqueName: \"kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:19.847479 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.847458 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4w7\" (UniqueName: \"kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.847545 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.847525 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmnj\" (UniqueName: \"kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.847591 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.847572 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwl4\" (UniqueName: \"kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.847622 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.847602 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njcm\" (UniqueName: \"kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.848727 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.848709 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llh49\" (UniqueName: \"kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:42:19.848890 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.848877 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lbv\" (UniqueName: \"kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:19.964097 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.964068 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:42:19.972139 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.972116 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:42:19.972926 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.972903 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6266e78f_7ca5_43bf_b3e6_6206fc45c4c9.slice/crio-989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90 WatchSource:0}: Error finding container 989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90: Status 404 returned error can't find the container with id 989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90 Oct 30 00:42:19.980505 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.980482 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode469533b_597a_4cf2_afd5_ad2fa146a978.slice/crio-32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152 WatchSource:0}: Error finding container 32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152: Status 404 returned error can't find the container with id 32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152 Oct 30 00:42:19.980606 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.980557 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:42:19.988334 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.988315 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:42:19.988588 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.988566 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2abc6d_480c_4b36_a525_ab1e04bf57c0.slice/crio-bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347 WatchSource:0}: Error finding container bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347: Status 404 returned error can't find the container with id bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347 Oct 30 00:42:19.989245 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:19.989231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:42:19.996679 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.996660 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dd9e0f_977b_454c_8599_f6357642c0b6.slice/crio-7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a WatchSource:0}: Error finding container 7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a: Status 404 returned error can't find the container with id 7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a Oct 30 00:42:19.998876 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:19.998837 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043e9d07_b27e_4378_b055_c4d5bdcc29e3.slice/crio-a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9 WatchSource:0}: Error finding container a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9: Status 404 returned error can't find the container with id a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9 Oct 30 00:42:20.001072 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.001052 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:20.002268 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.002251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7tv9" Oct 30 00:42:20.007481 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.007461 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:20.010871 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:20.010839 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b428a8_331e_43e7_bfc1_fe7e0e6732fe.slice/crio-34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76 WatchSource:0}: Error finding container 34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76: Status 404 returned error can't find the container with id 34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76 Oct 30 00:42:20.010943 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.010925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:42:20.011842 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:20.011825 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fd86b2_89ac_4e07_8dd7_da1cf88785c3.slice/crio-a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22 WatchSource:0}: Error finding container a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22: Status 404 returned error can't find the container with id a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22 Oct 30 00:42:20.019357 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:20.019330 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368c928a_a45a_4446_942b_39bd53f20a04.slice/crio-923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a WatchSource:0}: Error finding container 923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a: Status 404 returned error can't find the container with id 923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a Oct 30 00:42:20.020494 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:42:20.020472 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f159ccf_7e4f_45d2_86e5_1b40e51df4b3.slice/crio-1a0d233dfaf5333df94cc0d0e4860115b289636f12e371958759033839a90103 WatchSource:0}: Error finding container 1a0d233dfaf5333df94cc0d0e4860115b289636f12e371958759033839a90103: Status 404 returned error can't find the container with id 1a0d233dfaf5333df94cc0d0e4860115b289636f12e371958759033839a90103 Oct 30 00:42:20.330767 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.330734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:20.330935 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.330782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:20.330997 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.330976 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:20.331063 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.331040 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:21.331024938 +0000 UTC m=+3.085844838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:20.331521 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.331498 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:20.331640 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.331565 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:21.331548621 +0000 UTC m=+3.086368519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:20.432852 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.432172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:20.432852 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.432375 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:20.432852 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.432401 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:20.432852 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.432414 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:20.432852 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:20.432477 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:21.432457699 +0000 UTC m=+3.187277606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:20.454510 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.454466 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:20.540655 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.540514 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:20.721480 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.721269 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" event={"ID":"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3","Type":"ContainerStarted","Data":"1a0d233dfaf5333df94cc0d0e4860115b289636f12e371958759033839a90103"} Oct 30 00:42:20.726140 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.726104 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ts2gs" event={"ID":"368c928a-a45a-4446-942b-39bd53f20a04","Type":"ContainerStarted","Data":"923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a"} Oct 30 00:42:20.741027 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.740959 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22"} Oct 30 00:42:20.743596 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.743523 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7tv9" event={"ID":"54b428a8-331e-43e7-bfc1-fe7e0e6732fe","Type":"ContainerStarted","Data":"34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76"} Oct 30 00:42:20.748362 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.748251 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mst6g" event={"ID":"043e9d07-b27e-4378-b055-c4d5bdcc29e3","Type":"ContainerStarted","Data":"a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9"} Oct 30 00:42:20.753011 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.752921 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" event={"ID":"7f2abc6d-480c-4b36-a525-ab1e04bf57c0","Type":"ContainerStarted","Data":"bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347"} Oct 30 00:42:20.757686 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.757607 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerStarted","Data":"7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a"} Oct 30 00:42:20.760839 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.760770 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kp9nh" event={"ID":"e469533b-597a-4cf2-afd5-ad2fa146a978","Type":"ContainerStarted","Data":"32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152"} Oct 30 00:42:20.765508 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.765410 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:20 +0000 UTC" deadline="2028-06-04 10:41:34.696718008 +0000 UTC" Oct 30 00:42:20.765508 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.765460 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14001h59m13.931262338s" Oct 30 00:42:20.767496 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:20.767434 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6qb5x" event={"ID":"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9","Type":"ContainerStarted","Data":"989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90"} Oct 30 00:42:21.341046 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.341008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:21.341240 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.341064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:21.341240 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.341191 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:21.341240 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.341192 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:21.341418 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.341262 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:23.341243038 +0000 UTC m=+5.096062944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:21.341418 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.341292 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:23.341280791 +0000 UTC m=+5.096100686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:21.441888 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.441844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:21.442094 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.442059 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:21.442094 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.442080 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:21.442094 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.442095 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:21.442269 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.442154 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:23.442136153 +0000 UTC m=+5.196956057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:21.693476 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.693440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:21.693675 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.693591 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:21.694052 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.694033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:21.694161 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.694133 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:21.694248 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.694211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:21.694310 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:21.694290 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:21.766017 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.765965 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:20 +0000 UTC" deadline="2028-08-15 10:45:38.460956748 +0000 UTC" Oct 30 00:42:21.766017 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:21.766015 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15730h3m16.694947361s" Oct 30 00:42:23.359033 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.358721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:23.359516 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.359064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:23.359516 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.359236 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:23.359516 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.359305 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.359286197 +0000 UTC m=+9.114106099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:23.359728 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.359709 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:23.359803 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.359769 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.359752827 +0000 UTC m=+9.114572731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:23.459803 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.459759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:23.460052 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.460032 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:23.460153 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.460063 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:23.460153 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.460078 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:23.460257 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.460156 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:27.460136206 +0000 UTC m=+9.214956108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:23.694467 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.694295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:23.694467 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.694443 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:23.694910 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.694868 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:23.695030 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.694995 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:23.695093 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:23.695072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:23.695207 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:23.695162 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:24.796805 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.796050 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" event={"ID":"09bade25a7c8ebe0a0dcc7613db38300","Type":"ContainerStarted","Data":"c26bdc560d2f63cc1c604eb06cd502f31cfc42d6ca6de980dc0eedd82d324f10"} Oct 30 00:42:24.806611 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.806088 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ts2gs" event={"ID":"368c928a-a45a-4446-942b-39bd53f20a04","Type":"ContainerStarted","Data":"165bb610182bb70b9e72224022efaaa3c9445134bce1e9198905db6db5cfe440"} Oct 30 00:42:24.810429 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.810215 2572 generic.go:356] "Generic (PLEG): container finished" podID="767c439e26e5517d9df3fdb3ccab0ad7" containerID="0e35f58ccf5ba45b2940361acd67815919eea34db04e6beccaddc0cfbb7cb3df" exitCode=0 Oct 30 00:42:24.810429 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.810335 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerDied","Data":"0e35f58ccf5ba45b2940361acd67815919eea34db04e6beccaddc0cfbb7cb3df"} Oct 30 00:42:24.818239 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.817583 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="f3924cf0f094da073906d53918c4d49de555f64206c60f001b3a658d2c9d3a94" exitCode=0 Oct 30 00:42:24.818239 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.817628 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"f3924cf0f094da073906d53918c4d49de555f64206c60f001b3a658d2c9d3a94"} Oct 30 00:42:24.893559 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.893472 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" podStartSLOduration=5.893451772 podStartE2EDuration="5.893451772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:24.851250207 +0000 UTC m=+6.606070120" watchObservedRunningTime="2026-10-30 00:42:24.893451772 +0000 UTC m=+6.648271686" Oct 30 00:42:24.991967 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:24.989544 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ts2gs" podStartSLOduration=1.457284209 podStartE2EDuration="5.989522366s" totalImagesPullingTime="4.532238157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:24.985561879 +0000 UTC m=+6.740381790" watchObservedRunningTime="2026-10-30 00:42:24.989522366 +0000 UTC m=+6.744342279" Oct 30 00:42:25.693732 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.693704 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:25.693915 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:25.693889 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:25.694293 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.694275 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:25.694412 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:25.694391 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:25.694466 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.694458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:25.694552 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:25.694535 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:25.830523 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.830441 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerStarted","Data":"46e1954b54de11635dd1a4269d45dc6f95dd1cfe859a8d92a354bf7594436ce1"} Oct 30 00:42:25.834898 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.834758 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kp9nh" event={"ID":"e469533b-597a-4cf2-afd5-ad2fa146a978","Type":"ContainerStarted","Data":"c8d12538c382a1c599e0ccda7ceea0305fb16e093a1f410dca5c7de5473332bf"} Oct 30 00:42:25.912029 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:25.911965 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" podStartSLOduration=6.91194734 podStartE2EDuration="6.91194734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:25.87205512 +0000 UTC m=+7.626875030" watchObservedRunningTime="2026-10-30 00:42:25.91194734 +0000 UTC m=+7.666767252" Oct 30 00:42:26.839909 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:26.839820 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" event={"ID":"7f2abc6d-480c-4b36-a525-ab1e04bf57c0","Type":"ContainerStarted","Data":"18b131d316f23aab775e31de850ca22a8e33aa9a5461858943fc034764a46d63"} Oct 30 00:42:27.393039 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.392994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:27.393238 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.393058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:27.393238 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.393227 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:27.393358 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.393291 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:35.393271787 +0000 UTC m=+17.148091697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:27.393697 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.393677 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:27.393797 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.393732 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:35.393717113 +0000 UTC m=+17.148537015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:27.493891 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.493796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:27.494052 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.494013 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:27.494052 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.494032 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:27.494052 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.494043 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:27.494200 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.494103 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:35.494084203 +0000 UTC m=+17.248904110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:27.693687 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.693645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:27.693849 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.693789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:27.694205 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.694185 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:27.694314 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.694270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:27.694314 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.694308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:27.694448 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:27.694361 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:27.961072 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:27.961020 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:42:28.697255 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.694174 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:42:27.961039777Z","UUID":"3fee9155-a1c8-4357-b3bf-baaf54170349","Handler":null,"Name":"","Endpoint":""} Oct 30 00:42:28.698161 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.698129 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:42:28.698587 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.698167 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:42:28.849363 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.849117 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7tv9" event={"ID":"54b428a8-331e-43e7-bfc1-fe7e0e6732fe","Type":"ContainerStarted","Data":"fd03a5966aa9a3eb378552dd91aa9c9e8b881a3ab3bf14e7e0f22509840b35ba"} Oct 30 00:42:28.851752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.851697 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" event={"ID":"7f2abc6d-480c-4b36-a525-ab1e04bf57c0","Type":"ContainerStarted","Data":"30f1291bc6b91584d5c6a6598509bc291e031848a9868ecb46321d40b1544ff9"} Oct 30 00:42:28.890907 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.890401 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g7tv9" podStartSLOduration=1.51154861 podStartE2EDuration="9.890386825s" totalImagesPullingTime="8.378838215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:28.890069655 +0000 UTC m=+10.644889565" watchObservedRunningTime="2026-10-30 00:42:28.890386825 +0000 UTC m=+10.645206735" Oct 30 00:42:28.890907 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:28.890650 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kp9nh" podStartSLOduration=4.342522345 podStartE2EDuration="9.890646414s" totalImagesPullingTime="5.548124069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:25.912411383 +0000 UTC m=+7.667231295" watchObservedRunningTime="2026-10-30 00:42:28.890646414 +0000 UTC m=+10.645466322" Oct 30 00:42:29.693459 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.693432 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:29.693981 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:29.693561 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:29.693981 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.693945 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:29.694058 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:29.694034 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:29.694117 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.694086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:29.694185 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:29.694153 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:29.792079 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.792048 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:29.792691 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.792668 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:29.856731 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.856705 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" event={"ID":"7f2abc6d-480c-4b36-a525-ab1e04bf57c0","Type":"ContainerStarted","Data":"af53d979fa058360968b39b9dd2a53d148bd9e3a0e610a580961ebc11f4e4e1b"} Oct 30 00:42:29.859280 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.859230 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="671544b72508f6b995de303c3137458b8befc67078ff5d42f64d236a55f95f6b" exitCode=0 Oct 30 00:42:29.859402 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.859313 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"671544b72508f6b995de303c3137458b8befc67078ff5d42f64d236a55f95f6b"} Oct 30 00:42:29.861232 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.861208 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6qb5x" event={"ID":"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9","Type":"ContainerStarted","Data":"a2c031590e694e342bb4d117a4ca655af6d392195d738cad47d036bf3c9bcd1b"} Oct 30 00:42:29.940949 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.940891 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" podStartSLOduration=2.280565488 podStartE2EDuration="10.940874631s" totalImagesPullingTime="8.660309143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:29.89937268 +0000 UTC m=+11.654192590" watchObservedRunningTime="2026-10-30 00:42:29.940874631 +0000 UTC m=+11.695694531" Oct 30 00:42:29.983810 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:29.983696 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6qb5x" podStartSLOduration=2.204176932 podStartE2EDuration="10.983677538s" totalImagesPullingTime="8.779500606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:29.941256372 +0000 UTC m=+11.696076280" watchObservedRunningTime="2026-10-30 00:42:29.983677538 +0000 UTC m=+11.738497448" Oct 30 00:42:30.865688 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.865655 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" event={"ID":"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3","Type":"ContainerStarted","Data":"e8eec649aaa4d48e9e37cb6b68eb623463bf673e1e616c81c1e08b1009213d73"} Oct 30 00:42:30.868625 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.868598 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mst6g" event={"ID":"043e9d07-b27e-4378-b055-c4d5bdcc29e3","Type":"ContainerStarted","Data":"fe953631129763fe695c08d1ae8d65b2962f841ee94d4146f2204de9863aacbe"} Oct 30 00:42:30.871175 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.871142 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="c65be92ffaffff89582e776efa64e461f46f5a71ae0b22a6a02710afcbc10626" exitCode=0 Oct 30 00:42:30.871281 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.871181 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"c65be92ffaffff89582e776efa64e461f46f5a71ae0b22a6a02710afcbc10626"} Oct 30 00:42:30.906899 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.906828 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" podStartSLOduration=1.193122871 podStartE2EDuration="11.906810588s" totalImagesPullingTime="10.713687717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.906800448 +0000 UTC m=+12.661620357" watchObservedRunningTime="2026-10-30 00:42:30.906810588 +0000 UTC m=+12.661630478" Oct 30 00:42:30.992925 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:30.992807 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mst6g" podStartSLOduration=3.535102921 podStartE2EDuration="11.992794844s" totalImagesPullingTime="8.457691923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.992711531 +0000 UTC m=+12.747531439" watchObservedRunningTime="2026-10-30 00:42:30.992794844 +0000 UTC m=+12.747614753" Oct 30 00:42:31.693561 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:31.693523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:31.693561 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:31.693557 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:31.693774 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:31.693578 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:31.693774 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:31.693695 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:31.693830 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:31.693782 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:31.693882 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:31.693851 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:32.877431 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:32.877242 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="d2fe78c01d80250f724890c147f4fe84c8dbc67cdfefaac80f990dedcefda717" exitCode=0 Oct 30 00:42:32.877431 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:32.877323 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"d2fe78c01d80250f724890c147f4fe84c8dbc67cdfefaac80f990dedcefda717"} Oct 30 00:42:33.693678 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:33.693648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:33.694130 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:33.693748 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:33.694130 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:33.693790 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:33.694130 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:33.693847 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:33.694130 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:33.693963 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:33.694130 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:33.694070 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:35.451360 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.451205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:35.451360 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.451351 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:35.451803 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.451376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:35.451803 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.451398 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:51.451385058 +0000 UTC m=+33.206204945 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:35.451803 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.451441 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:35.451803 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.451479 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:51.451469023 +0000 UTC m=+33.206288910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:35.552512 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.552461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:35.552638 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.552618 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:35.552711 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.552644 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:35.552711 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.552657 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:35.552711 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.552707 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:51.552694428 +0000 UTC m=+33.307514318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:35.694201 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.694175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:35.694201 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.694200 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:35.694459 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.694270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:35.694459 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.694349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:35.694459 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.694451 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:35.694648 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:35.694558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:35.886514 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886443 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:42:35.886771 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886751 2572 generic.go:356] "Generic (PLEG): container finished" podID="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" containerID="9a330014799fd7213a2e5ecbd8311ca93c8c6f953c5aada935e123a222b2d6e7" exitCode=1 Oct 30 00:42:35.886838 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886820 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"f3f8137ef45ef3619e47638dac6d8dc5dee61a414d837b811645e8a59378b020"} Oct 30 00:42:35.886912 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886904 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"ab3c38918a592d304c2c1efd4c2a281792478eb8af3b4b72037d43b9cd6bbee2"} Oct 30 00:42:35.886959 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886946 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"afd6607cec38706ecda8da5991586823e83ea4344991262a88b36e45d1ae6e71"} Oct 30 00:42:35.887005 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.886987 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"7e30edd57d40e46b88773de4b122ae9686e5db4f6e7c6a0563bbfd5939f1589a"} Oct 30 00:42:35.887069 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.887055 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerDied","Data":"9a330014799fd7213a2e5ecbd8311ca93c8c6f953c5aada935e123a222b2d6e7"} Oct 30 00:42:35.887121 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.887108 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"8b3978ae7409ec2bd802f8ec0dbbe310c6bff4d844e883e49ac2c77401bd79a5"} Oct 30 00:42:35.889502 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.889478 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="49b71e7c5c2cf1e568ebd213830dd3726cc2510fc854dd733050b56a8d1a64b3" exitCode=0 Oct 30 00:42:35.889616 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:35.889515 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"49b71e7c5c2cf1e568ebd213830dd3726cc2510fc854dd733050b56a8d1a64b3"} Oct 30 00:42:36.894828 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:36.894795 2572 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="3b2a19b30c3ab9f76f0eb4d888fd2219a10599e0121cfe50a47f42f550141e16" exitCode=0 Oct 30 00:42:36.895505 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:36.894849 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerDied","Data":"3b2a19b30c3ab9f76f0eb4d888fd2219a10599e0121cfe50a47f42f550141e16"} Oct 30 00:42:37.076993 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.076969 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6qb5x_6266e78f-7ca5-43bf-b3e6-6206fc45c4c9/dns-node-resolver/0.log" Oct 30 00:42:37.693411 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.693381 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:37.693555 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.693493 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:37.693555 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:37.693505 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:37.693555 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.693524 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:37.693662 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:37.693609 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:37.693695 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:37.693665 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:37.900081 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.900059 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:42:37.900536 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.900453 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"837b018919be4b102ba82170388621453211e2bf50d603ae91dca844924f1c18"} Oct 30 00:42:37.903634 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.903614 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k98jz" event={"ID":"49dd9e0f-977b-454c-8599-f6357642c0b6","Type":"ContainerStarted","Data":"c89423b88a613bdac450fbf820eb2765e3fcf7fdbfd10169ef385b9ee0986980"} Oct 30 00:42:37.948809 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:37.948752 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k98jz" podStartSLOduration=7.74900607 podStartE2EDuration="18.948737092s" totalImagesPullingTime="11.199731022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:37.948513122 +0000 UTC m=+19.703333030" watchObservedRunningTime="2026-10-30 00:42:37.948737092 +0000 UTC m=+19.703557000" Oct 30 00:42:38.030934 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:38.030905 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kp9nh_e469533b-597a-4cf2-afd5-ad2fa146a978/node-ca/0.log" Oct 30 00:42:38.557407 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:38.557378 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:38.557579 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:38.557489 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:42:38.557926 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:38.557912 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:42:39.693390 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.693361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:39.693390 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.693370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:39.693902 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:39.693451 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:39.693902 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.693506 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:39.693902 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:39.693628 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:39.693902 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:39.693738 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:39.912034 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.911849 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:42:39.912376 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.912353 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"f41f5b05cdc94594e6632aae1e50c4311e9730c8f4af6b27bf4d6257fcb39ca8"} Oct 30 00:42:39.912650 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.912627 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:39.912717 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.912710 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:39.912811 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.912796 2572 scope.go:122] "RemoveContainer" containerID="9a330014799fd7213a2e5ecbd8311ca93c8c6f953c5aada935e123a222b2d6e7" Oct 30 00:42:39.928676 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:39.928655 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:40.918261 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:40.918234 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:42:40.918731 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:40.918640 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" event={"ID":"69fd86b2-89ac-4e07-8dd7-da1cf88785c3","Type":"ContainerStarted","Data":"539c8bc894725ac5661aa4a4b0448cf36dd6f79ea0f4abca9e07c0623ae46e7d"} Oct 30 00:42:40.918830 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:40.918815 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:40.934186 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:40.934158 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:42:41.008674 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:41.008627 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" podStartSLOduration=6.934176517 podStartE2EDuration="22.008616219s" totalImagesPullingTime="15.074439702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:40.994665774 +0000 UTC m=+22.749485682" watchObservedRunningTime="2026-10-30 00:42:41.008616219 +0000 UTC m=+22.763436127" Oct 30 00:42:41.693150 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:41.693122 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:41.693150 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:41.693146 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:41.693415 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:41.693215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:41.693415 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:41.693302 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:41.693415 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:41.693405 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:41.693527 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:41.693458 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:43.694023 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:43.693995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:43.694451 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:43.693995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:43.694451 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:43.694100 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:43.694451 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:43.694195 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:43.694451 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:43.694001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:43.694451 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:43.694294 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:45.694106 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:45.694078 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:45.694554 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:45.694169 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:45.694554 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:45.694185 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:45.694554 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:45.694272 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:45.694554 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:45.694356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:45.694554 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:45.694443 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:47.693468 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:47.693436 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:47.693468 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:47.693452 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:47.693892 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:47.693446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:47.693892 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:47.693535 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:47.693892 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:47.693594 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:47.693892 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:47.693651 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:49.693727 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:49.693697 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:49.694163 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:49.693772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:49.694163 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:49.693798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:49.694163 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:49.693893 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:49.694163 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:49.693956 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:49.694163 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:49.694054 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:51.457588 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.457555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:51.457588 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.457591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:51.458133 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.457696 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:51.458133 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.457713 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:51.458133 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.457752 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret podName:1ab30191-af9f-4578-ae26-c57232aa5111 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:23.457736977 +0000 UTC m=+65.212556871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret") pod "global-pull-secret-syncer-ncxvc" (UID: "1ab30191-af9f-4578-ae26-c57232aa5111") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:51.458133 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.457767 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs podName:d7d248e8-94d3-40dd-baf3-9cfb15053504 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:23.457760313 +0000 UTC m=+65.212580200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs") pod "network-metrics-daemon-mrgqt" (UID: "d7d248e8-94d3-40dd-baf3-9cfb15053504") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:51.558763 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.558729 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:51.558940 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.558843 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:51.558940 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.558875 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:51.558940 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.558888 2572 projected.go:196] Error preparing data for projected volume kube-api-access-vtjfh for pod openshift-network-diagnostics/network-check-target-x9kzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:51.558940 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.558933 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh podName:ed92d686-6ba7-49de-b947-7178cb50b3c0 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:23.558921782 +0000 UTC m=+65.313741682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtjfh" (UniqueName: "kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh") pod "network-check-target-x9kzn" (UID: "ed92d686-6ba7-49de-b947-7178cb50b3c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:51.693372 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.693339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:51.693550 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.693377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:51.693550 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:51.693377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:51.693550 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.693466 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:51.693550 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.693526 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:51.693759 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:51.693585 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:53.694137 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:53.694106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:53.694518 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:53.694189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:53.694518 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:53.694281 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:53.694518 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:53.694290 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:53.694518 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:53.694376 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:53.694518 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:53.694432 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:55.693092 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:55.693062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:55.693563 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:55.693173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:55.693563 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:55.693194 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:55.693563 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:55.693171 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:55.693563 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:55.693243 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:55.693563 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:55.693343 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:57.694181 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:57.694146 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:57.694651 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:57.694285 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:57.694651 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:57.694298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:57.694651 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:57.694310 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:57.694651 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:57.694356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:57.694651 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:57.694406 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:42:59.694011 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:59.693980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:42:59.694399 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:59.694077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:42:59.694399 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:59.694088 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:42:59.694399 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:59.694157 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:42:59.694399 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:42:59.694161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:42:59.694399 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:42:59.694218 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:43:00.839987 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.839730 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x9kzn"] Oct 30 00:43:00.840581 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.840560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:00.840912 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:00.840853 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:43:00.844908 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.844880 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mrgqt"] Oct 30 00:43:00.845009 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.844990 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncxvc"] Oct 30 00:43:00.845093 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.845079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:00.845232 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:00.845210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:43:00.845305 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:00.845292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:00.845423 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:00.845407 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:43:02.693146 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:02.693117 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:02.693146 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:02.693152 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:02.693697 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:02.693158 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:02.693697 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:02.693259 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:43:02.693697 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:02.693367 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:43:02.693697 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:02.693433 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:43:04.693840 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:04.693811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:04.694299 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:04.693812 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:04.694299 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:04.693952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:04.694299 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:04.693947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x9kzn" podUID="ed92d686-6ba7-49de-b947-7178cb50b3c0" Oct 30 00:43:04.694299 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:04.694029 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncxvc" podUID="1ab30191-af9f-4578-ae26-c57232aa5111" Oct 30 00:43:04.694299 ip-10-0-143-7 kubenswrapper[2572]: E1030 00:43:04.694203 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrgqt" podUID="d7d248e8-94d3-40dd-baf3-9cfb15053504" Oct 30 00:43:05.026478 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.026405 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeReady" Oct 30 00:43:05.026624 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.026536 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:43:05.077953 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.077921 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8n2dz"] Oct 30 00:43:05.079949 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.079934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.081925 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.081896 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gz2md"] Oct 30 00:43:05.084278 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.084257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.099984 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.099961 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8n2dz"] Oct 30 00:43:05.103551 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.103530 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:43:05.103733 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.103719 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:43:05.104820 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.104795 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7kjw6"] Oct 30 00:43:05.106636 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.106617 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gz2md"] Oct 30 00:43:05.106736 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.106721 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.113010 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.112981 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:43:05.113010 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.112990 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:43:05.113133 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.113031 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\"" Oct 30 00:43:05.113133 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.113064 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\"" Oct 30 00:43:05.113133 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.113077 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:43:05.130676 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.130656 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kjw6"] Oct 30 00:43:05.151529 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.151501 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24vm\" (UniqueName: \"kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.151627 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.151540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.151627 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.151566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.151699 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.151679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.202716 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.202685 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:43:05.202878 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.202745 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:43:05.202878 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.202790 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:43:05.203079 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.203062 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\"" Oct 30 00:43:05.203150 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.203134 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:43:05.252460 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.252460 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252460 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.252646 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.252646 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.252737 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.252737 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.252833 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.252833 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndl8\" (UniqueName: \"kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.252833 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252823 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.253006 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rqg\" (UniqueName: \"kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.253006 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.253006 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.252987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24vm\" (UniqueName: \"kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.253239 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.253220 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.257026 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.257010 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.273317 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.273295 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24vm\" (UniqueName: \"kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.353372 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rqg\" (UniqueName: \"kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.353372 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353372 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353361 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.353565 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353565 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353565 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353454 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353565 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353481 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353565 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndl8\" (UniqueName: \"kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.353820 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.353794 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.354042 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.354020 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.355832 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.355807 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.355905 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.355894 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.373573 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.373547 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rqg\" (UniqueName: \"kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.373661 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.373555 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndl8\" (UniqueName: \"kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.396032 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.396007 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:05.399243 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.399226 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:05.418063 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.418040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:05.713276 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.713225 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gz2md"] Oct 30 00:43:05.716378 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.716354 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8n2dz"] Oct 30 00:43:05.723435 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:05.723410 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ed2301_70f6_497f_b717_be08bdb686c4.slice/crio-69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f WatchSource:0}: Error finding container 69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f: Status 404 returned error can't find the container with id 69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f Oct 30 00:43:05.744400 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.744379 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kjw6"] Oct 30 00:43:05.745367 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:05.745348 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c43657_fe00_4520_9ee8_684223518dec.slice/crio-fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e WatchSource:0}: Error finding container fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e: Status 404 returned error can't find the container with id fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e Oct 30 00:43:05.968965 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.968884 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kjw6" event={"ID":"a7c43657-fe00-4520-9ee8-684223518dec","Type":"ContainerStarted","Data":"1cca3d10ec5f2771e6f25227712a9d2ec7bca885ff1fad5d1b64e296090b292a"} Oct 30 00:43:05.968965 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.968948 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kjw6" event={"ID":"a7c43657-fe00-4520-9ee8-684223518dec","Type":"ContainerStarted","Data":"fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e"} Oct 30 00:43:05.970038 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.970005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gz2md" event={"ID":"1437846a-8b02-48e2-8963-d04d2432ecae","Type":"ContainerStarted","Data":"4d738ef0005d5bace2e856b164fbee0f1fa92fd4623aefd675a46ccd344b957e"} Oct 30 00:43:05.971193 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:05.971172 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8n2dz" event={"ID":"58ed2301-70f6-497f-b717-be08bdb686c4","Type":"ContainerStarted","Data":"69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f"} Oct 30 00:43:06.695376 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.693265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:06.695376 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.693299 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:06.695376 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.693649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:06.711961 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.711899 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:06.725630 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.725604 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:06.726018 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.725729 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:06.726018 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.725754 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:06.726018 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.725954 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:06.726179 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.726060 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:06.731725 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.731704 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:06.823201 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.823169 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm"] Oct 30 00:43:06.825786 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.825766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:06.845746 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.845721 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:43:06.845905 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.845883 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:43:06.846009 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.845991 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 00:43:06.846066 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.846040 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:43:06.846066 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.846061 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:43:06.846146 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.846062 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6vzmx\"" Oct 30 00:43:06.852803 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.852776 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm"] Oct 30 00:43:06.857420 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.857393 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf"] Oct 30 00:43:06.859708 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.859687 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.880279 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.880258 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8mh62"] Oct 30 00:43:06.882988 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.882631 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.891419 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.891398 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf"] Oct 30 00:43:06.946048 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.945972 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 00:43:06.946295 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946209 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:43:06.946425 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946369 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:43:06.946425 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946402 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 00:43:06.946553 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946453 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m82nk\"" Oct 30 00:43:06.946553 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946490 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:43:06.946553 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946510 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:43:06.946553 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.946453 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbt7f\"" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965523 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zbd\" (UniqueName: \"kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbsk6\" (UniqueName: \"kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965750 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqhg\" (UniqueName: \"kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.966309 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:06.967264 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:06.967264 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965949 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.967264 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.967264 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.965988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:06.977033 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:06.976999 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kjw6" event={"ID":"a7c43657-fe00-4520-9ee8-684223518dec","Type":"ContainerStarted","Data":"208b23d29271520551c1c629af6c80dc75811b22babd227c9f33b8bc42b476ee"} Oct 30 00:43:07.066461 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqhg\" (UniqueName: \"kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.066742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zbd\" (UniqueName: \"kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbsk6\" (UniqueName: \"kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.066993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.067020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.067047 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.067138 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.067121 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068116 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.067684 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068236 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068133 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068236 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068188 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.068769 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068391 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068769 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068467 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068769 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068483 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.068769 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068623 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.069021 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.068778 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.069418 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.069396 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.070752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.070513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.070752 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.070704 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.071358 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.071337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.072384 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.072360 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.073031 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.072991 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.073593 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.073569 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.087520 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.087475 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqhg\" (UniqueName: \"kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.089150 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.089128 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbsk6\" (UniqueName: \"kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.090983 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.090941 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zbd\" (UniqueName: \"kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.140432 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.140396 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:07.175219 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.175190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:07.199918 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.199842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:07.218358 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:07.218330 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab413ab_1d2a_4e3d_bb80_b6a3e7a8fb8d.slice/crio-52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064 WatchSource:0}: Error finding container 52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064: Status 404 returned error can't find the container with id 52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064 Oct 30 00:43:07.460599 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.460562 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm"] Oct 30 00:43:07.462156 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:07.462125 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80b6429_b22b_4105_a70a_799faed22645.slice/crio-930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498 WatchSource:0}: Error finding container 930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498: Status 404 returned error can't find the container with id 930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498 Oct 30 00:43:07.495878 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.495838 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf"] Oct 30 00:43:07.505003 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:07.499379 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40834bbb_90ae_486f_bbcb_ff665c3f313e.slice/crio-0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75 WatchSource:0}: Error finding container 0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75: Status 404 returned error can't find the container with id 0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75 Oct 30 00:43:07.987434 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.987348 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" event={"ID":"40834bbb-90ae-486f-bbcb-ff665c3f313e","Type":"ContainerStarted","Data":"0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75"} Oct 30 00:43:07.991348 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.991313 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" event={"ID":"a80b6429-b22b-4105-a70a-799faed22645","Type":"ContainerStarted","Data":"f6b143313637195808e0e5d9fba7d988f2ec5a66ef3f5cf79c2c1a4d1b27664b"} Oct 30 00:43:07.991466 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.991379 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" event={"ID":"a80b6429-b22b-4105-a70a-799faed22645","Type":"ContainerStarted","Data":"e74267fb1fba9a5d23de85c161d9684619e2634a81d92827352e1b99b4c15831"} Oct 30 00:43:07.991466 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.991408 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" event={"ID":"a80b6429-b22b-4105-a70a-799faed22645","Type":"ContainerStarted","Data":"930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498"} Oct 30 00:43:07.998788 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.997958 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8n2dz" event={"ID":"58ed2301-70f6-497f-b717-be08bdb686c4","Type":"ContainerStarted","Data":"fbcc1803eb5f86b0a66e7bfff3c6bd3f9146e3b6d8898404f48b6d2ececd45d0"} Oct 30 00:43:07.998788 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.998010 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8n2dz" event={"ID":"58ed2301-70f6-497f-b717-be08bdb686c4","Type":"ContainerStarted","Data":"f9fa4d95dcb7fd28efd3895604d12f6261b4115d56085fb6c8719acec1e2a357"} Oct 30 00:43:07.998788 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:07.998648 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:08.001379 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.001353 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mh62" event={"ID":"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d","Type":"ContainerStarted","Data":"52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064"} Oct 30 00:43:08.045254 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.045197 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8n2dz" podStartSLOduration=1.055757937 podStartE2EDuration="3.045182634s" totalImagesPullingTime="1.989424697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:08.044256875 +0000 UTC m=+49.799076783" watchObservedRunningTime="2026-10-30 00:43:08.045182634 +0000 UTC m=+49.800002564" Oct 30 00:43:08.277971 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.277928 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:43:08.281626 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.281605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.315618 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.315590 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:43:08.315772 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.315733 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:43:08.315846 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.315805 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:43:08.316279 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.315913 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:43:08.316279 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.316012 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:43:08.316279 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.316073 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:43:08.316279 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.316246 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:43:08.317284 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.317170 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:43:08.317697 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.317575 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:43:08.320922 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.320879 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n682p\"" Oct 30 00:43:08.322571 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.322440 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:43:08.378746 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.378715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.378941 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.378760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.378941 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.378787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.378941 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.378811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379115 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.378941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379160 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379208 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379208 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379199 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379290 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379229 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379290 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379290 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379290 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.379490 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.379306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480384 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480524 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480550 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.481504 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.480578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.482171 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.481586 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.482221 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.482191 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.486837 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.486793 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.487970 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.487947 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.488050 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.488028 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.488989 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.488611 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.488989 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.488637 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.489231 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.489207 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.489338 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.489315 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.490417 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.490395 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.493474 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.493426 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.494988 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.494968 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.506093 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.506048 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.618361 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.618322 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:08.963036 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:08.962997 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:43:08.975946 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:08.975911 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66681562_5027_4a5d_9e34_762f99b60322.slice/crio-f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39 WatchSource:0}: Error finding container f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39: Status 404 returned error can't find the container with id f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39 Oct 30 00:43:09.006737 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.006702 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kjw6" event={"ID":"a7c43657-fe00-4520-9ee8-684223518dec","Type":"ContainerStarted","Data":"962aeb9e98a8a70488303b73c341da581b3848dc2913ced7561b25b0c9ce43c5"} Oct 30 00:43:09.009787 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.009332 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" event={"ID":"a80b6429-b22b-4105-a70a-799faed22645","Type":"ContainerStarted","Data":"6044af617f9c9d1f2da8051685ae3c5b1b3ad99b971b1c29dbdcb1894b2063b1"} Oct 30 00:43:09.011310 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.011285 2572 generic.go:356] "Generic (PLEG): container finished" podID="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" containerID="cf47c7cb6af786abc4cd4be7c11aac32d8e2250b055bdff1c7574a9ad08b158d" exitCode=0 Oct 30 00:43:09.011423 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.011369 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mh62" event={"ID":"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d","Type":"ContainerDied","Data":"cf47c7cb6af786abc4cd4be7c11aac32d8e2250b055bdff1c7574a9ad08b158d"} Oct 30 00:43:09.012735 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.012709 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39"} Oct 30 00:43:09.049164 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.049122 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7kjw6" podStartSLOduration=1.698536732 podStartE2EDuration="4.049107733s" totalImagesPullingTime="2.350571001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:43:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:09.04816839 +0000 UTC m=+50.802988298" watchObservedRunningTime="2026-10-30 00:43:09.049107733 +0000 UTC m=+50.803927641" Oct 30 00:43:09.126180 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.126143 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz"] Oct 30 00:43:09.136499 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.135310 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" podStartSLOduration=1.960217597 podStartE2EDuration="3.135293483s" totalImagesPullingTime="1.175075886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:09.13501613 +0000 UTC m=+50.889836039" watchObservedRunningTime="2026-10-30 00:43:09.135293483 +0000 UTC m=+50.890113393" Oct 30 00:43:09.153084 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.138699 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.157496 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.155438 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 00:43:09.157496 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.156832 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c35a6pei4g2kp\"" Oct 30 00:43:09.158584 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.157661 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 00:43:09.158584 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.157659 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz"] Oct 30 00:43:09.158584 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.157753 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 00:43:09.158584 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.157887 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b4t5b\"" Oct 30 00:43:09.158584 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.158047 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 00:43:09.255158 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.254958 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 00:43:09.292472 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.292448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.292628 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.292494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.292701 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.292684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.292751 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.292724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.292953 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.292918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.293050 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.293033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.293105 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.293088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.293206 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.293191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94j8\" (UniqueName: \"kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.293259 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.293224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.393927 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.393895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394168 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.393947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394168 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394168 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394168 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394168 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394426 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394426 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94j8\" (UniqueName: \"kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.394426 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.395089 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394700 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.395089 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.394897 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.397035 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.397009 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.397273 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.397252 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.397582 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.397557 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.397582 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.397568 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.397685 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.397675 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.398097 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.398080 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.415257 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.415217 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94j8\" (UniqueName: \"kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.493288 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.493251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:09.809194 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:09.809170 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz"] Oct 30 00:43:09.810576 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:09.810545 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeb58a8_a0fe_48f2_86a7_0e347a585522.slice/crio-faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015 WatchSource:0}: Error finding container faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015: Status 404 returned error can't find the container with id faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015 Oct 30 00:43:10.018500 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.018473 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" event={"ID":"40834bbb-90ae-486f-bbcb-ff665c3f313e","Type":"ContainerStarted","Data":"9f1a2ddcf2c914e322966a57051557ba1088bccbc5723e3b4452d7562d3723fd"} Oct 30 00:43:10.018876 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.018525 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" event={"ID":"40834bbb-90ae-486f-bbcb-ff665c3f313e","Type":"ContainerStarted","Data":"bceafec1ee913583a2a5321f0bb589d3b4ed729a8c5f997ec9de0936588adf5b"} Oct 30 00:43:10.018876 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.018552 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" event={"ID":"40834bbb-90ae-486f-bbcb-ff665c3f313e","Type":"ContainerStarted","Data":"ef1c0efdf716457928be37e8d8becf0ee7f7657ae2556498f9010a216370a51d"} Oct 30 00:43:10.019734 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.019713 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015"} Oct 30 00:43:10.021375 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.021345 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gz2md" event={"ID":"1437846a-8b02-48e2-8963-d04d2432ecae","Type":"ContainerStarted","Data":"ce008376b3f7ace67a444444d0f881fdb3d33ffac2452fa073dc8e18c6efd8e2"} Oct 30 00:43:10.023836 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.023812 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mh62" event={"ID":"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d","Type":"ContainerStarted","Data":"045dbfb65bdda27adf1e65650d7934cd42bf860d7da9dd48af96f0451e5acf2e"} Oct 30 00:43:10.023954 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.023882 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mh62" event={"ID":"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d","Type":"ContainerStarted","Data":"6f0d66f2188c4770d057eb2671d0dfaa06d0728060d6fb7e02dec2fad08ef98f"} Oct 30 00:43:10.061323 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.061238 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" podStartSLOduration=2.484392137 podStartE2EDuration="4.061219195s" totalImagesPullingTime="1.576827058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:10.060706634 +0000 UTC m=+51.815526542" watchObservedRunningTime="2026-10-30 00:43:10.061219195 +0000 UTC m=+51.816039104" Oct 30 00:43:10.106222 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.106174 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8mh62" podStartSLOduration=2.612925148 podStartE2EDuration="4.106157994s" totalImagesPullingTime="1.404427344s" totalInitContainerRuntime="88.805502ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:10.10480612 +0000 UTC m=+51.859626028" watchObservedRunningTime="2026-10-30 00:43:10.106157994 +0000 UTC m=+51.860977881" Oct 30 00:43:10.144592 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:10.144484 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gz2md" podStartSLOduration=1.891861983 podStartE2EDuration="5.144468218s" totalImagesPullingTime="3.252606235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:10.144001748 +0000 UTC m=+51.898821657" watchObservedRunningTime="2026-10-30 00:43:10.144468218 +0000 UTC m=+51.899288127" Oct 30 00:43:11.029222 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.029192 2572 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" exitCode=0 Oct 30 00:43:11.029667 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.029301 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba"} Oct 30 00:43:11.348577 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.348484 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f9c87f897-7zng6"] Oct 30 00:43:11.352068 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.352039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.364345 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.364275 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f9c87f897-7zng6"] Oct 30 00:43:11.368816 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.368786 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-sicahncopf47\"" Oct 30 00:43:11.368986 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.368836 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 00:43:11.368986 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.368935 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 00:43:11.369159 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.369004 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 00:43:11.369159 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.369108 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9p42h\"" Oct 30 00:43:11.369270 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.369163 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:43:11.515581 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.515821 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.515821 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515620 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdk5\" (UniqueName: \"kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.515821 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.515821 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.515821 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.516112 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.515823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616706 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdk5\" (UniqueName: \"kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.616919 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.617100 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.616935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.617557 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.617531 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.618081 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.618056 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.618404 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.618371 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.621244 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.621220 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.621397 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.621271 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.632406 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.632358 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.637464 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.637437 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdk5\" (UniqueName: \"kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.664949 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.664921 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:11.986079 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:11.986055 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f9c87f897-7zng6"] Oct 30 00:43:12.035108 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:12.035083 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"2c50218b09117bacf3cd3f397c127029e9fbe3959fde11b4d6bdbaa3e8a30c13"} Oct 30 00:43:12.035473 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:12.035164 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"3c8b4713a75b85be7a04a193ebf5c187002025c91191aff56abe10440bc5de53"} Oct 30 00:43:12.036444 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:12.036422 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" event={"ID":"ed7102eb-69ac-4caf-8039-18466d995097","Type":"ContainerStarted","Data":"986278046d4a44c9ab26f0be8ef21e3fc1accf36c41655f5904a1ff233fd4d17"} Oct 30 00:43:12.934257 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:12.934230 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:13.044619 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.044417 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5"} Oct 30 00:43:13.044619 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.044501 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083"} Oct 30 00:43:13.044619 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.044527 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563"} Oct 30 00:43:13.044619 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.044550 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538"} Oct 30 00:43:13.044619 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.044573 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d"} Oct 30 00:43:13.050039 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.050002 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"3bd748adb1f3e19b91d71331f79ba445321c1599266cb6d22a4583dad05a7c4d"} Oct 30 00:43:13.050132 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.050086 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"62a11057569776dc15c29e519722ae38a3c64c19ff9019e97b4a1108a730486f"} Oct 30 00:43:13.050178 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:13.050129 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"c52ebf31536d91333ba04737656a0f44b8612a22f8741a3df189d977aceafce5"} Oct 30 00:43:14.054769 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.054728 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" event={"ID":"ed7102eb-69ac-4caf-8039-18466d995097","Type":"ContainerStarted","Data":"b8547f06b88fc23216b301b25b4cf0885a3c93a3e080cd2e252d4179300b1f30"} Oct 30 00:43:14.057760 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.057735 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" event={"ID":"7eeb58a8-a0fe-48f2-86a7-0e347a585522","Type":"ContainerStarted","Data":"4ec4807c6098a8eef836ee4ad22097bb05fac87926d88a5440078e19c23b5233"} Oct 30 00:43:14.057891 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.057795 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:14.061090 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.061070 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerStarted","Data":"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba"} Oct 30 00:43:14.100090 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.100049 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" podStartSLOduration=1.5934647100000001 podStartE2EDuration="3.100036185s" totalImagesPullingTime="1.506571475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.099358184 +0000 UTC m=+55.854178094" watchObservedRunningTime="2026-10-30 00:43:14.100036185 +0000 UTC m=+55.854856093" Oct 30 00:43:14.159722 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.159677 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.44723568 podStartE2EDuration="6.159662643s" totalImagesPullingTime="2.698317661s" totalInitContainerRuntime="14.109302ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.159322754 +0000 UTC m=+55.914142670" watchObservedRunningTime="2026-10-30 00:43:14.159662643 +0000 UTC m=+55.914482530" Oct 30 00:43:14.208985 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:14.208926 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" podStartSLOduration=2.347626636 podStartE2EDuration="5.208907717s" totalImagesPullingTime="2.861281081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.206718656 +0000 UTC m=+55.961538565" watchObservedRunningTime="2026-10-30 00:43:14.208907717 +0000 UTC m=+55.963727627" Oct 30 00:43:19.026548 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:19.026517 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:20.070581 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:20.070556 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:23.506224 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.506160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:23.506224 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.506230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:23.521637 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.521612 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:23.529135 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.529103 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:23.535198 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.535180 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:23.539348 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.539325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:23.607172 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.607142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:23.622292 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.622267 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:23.643311 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.643289 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:23.651742 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.651719 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:23.825386 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.825303 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:23.830589 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.830572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:23.838547 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.838531 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:23.843436 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.843419 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:23.844411 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.844393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:23.849328 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:23.849306 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:24.126910 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:24.126890 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mrgqt"] Oct 30 00:43:24.127929 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:24.127903 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d248e8_94d3_40dd_baf3_9cfb15053504.slice/crio-84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719 WatchSource:0}: Error finding container 84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719: Status 404 returned error can't find the container with id 84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719 Oct 30 00:43:24.148480 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:24.147529 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncxvc"] Oct 30 00:43:24.148734 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:24.148714 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab30191_af9f_4578_ae26_c57232aa5111.slice/crio-8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61 WatchSource:0}: Error finding container 8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61: Status 404 returned error can't find the container with id 8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61 Oct 30 00:43:24.160790 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:24.160766 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x9kzn"] Oct 30 00:43:24.161697 ip-10-0-143-7 kubenswrapper[2572]: W1030 00:43:24.161671 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded92d686_6ba7_49de_b947_7178cb50b3c0.slice/crio-7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e WatchSource:0}: Error finding container 7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e: Status 404 returned error can't find the container with id 7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e Oct 30 00:43:25.102470 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:25.102433 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncxvc" event={"ID":"1ab30191-af9f-4578-ae26-c57232aa5111","Type":"ContainerStarted","Data":"8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61"} Oct 30 00:43:25.104894 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:25.104721 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x9kzn" event={"ID":"ed92d686-6ba7-49de-b947-7178cb50b3c0","Type":"ContainerStarted","Data":"7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e"} Oct 30 00:43:25.106652 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:25.106624 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrgqt" event={"ID":"d7d248e8-94d3-40dd-baf3-9cfb15053504","Type":"ContainerStarted","Data":"84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719"} Oct 30 00:43:26.120665 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:26.120613 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrgqt" event={"ID":"d7d248e8-94d3-40dd-baf3-9cfb15053504","Type":"ContainerStarted","Data":"7239e02c7c4cd1f712fff19de6187068349afa9bdc2f923db985197c564353b0"} Oct 30 00:43:26.121190 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:26.120713 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrgqt" event={"ID":"d7d248e8-94d3-40dd-baf3-9cfb15053504","Type":"ContainerStarted","Data":"bc4a480275f2f9334480d13a175e2cfc1da25a97e551b058e1f90832f1fdfda8"} Oct 30 00:43:26.161589 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:26.161533 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mrgqt" podStartSLOduration=66.129631913 podStartE2EDuration="1m7.1615205s" totalImagesPullingTime="1.031888587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:26.161044232 +0000 UTC m=+67.915864141" watchObservedRunningTime="2026-10-30 00:43:26.1615205 +0000 UTC m=+67.916340409" Oct 30 00:43:28.131543 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:28.131504 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x9kzn" event={"ID":"ed92d686-6ba7-49de-b947-7178cb50b3c0","Type":"ContainerStarted","Data":"7f71630a5ed248974fd7ef55f3e85c7f62f401ebd0409158f7ec0f3e5f449371"} Oct 30 00:43:28.132052 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:28.131594 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:28.171615 ip-10-0-143-7 kubenswrapper[2572]: I1030 00:43:28.171559 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x9kzn" podStartSLOduration=66.023054703 podStartE2EDuration="1m9.171543778s" totalImagesPullingTime="3.148489075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:28.170899938 +0000 UTC m=+69.925719856" watchObservedRunningTime="2026-10-30 00:43:28.171543778 +0000 UTC m=+69.926363686" Oct 30 00:43:29.094785 ip-10-0-143-7 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:43:29.115236 ip-10-0-143-7 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:43:29.115406 ip-10-0-143-7 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:43:29.115456 ip-10-0-143-7 systemd[1]: kubelet.service: Consumed 4.068s CPU time. Oct 30 00:43:29.126445 ip-10-0-143-7 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:43:29.186070 ip-10-0-143-7 kubenswrapper[5548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:29.186070 ip-10-0-143-7 kubenswrapper[5548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:43:29.186070 ip-10-0-143-7 kubenswrapper[5548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:29.186070 ip-10-0-143-7 kubenswrapper[5548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:29.188157 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188083 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.188157 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188158 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188162 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188166 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188179 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188182 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188185 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188188 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188191 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188193 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188196 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188198 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188201 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188203 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188206 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188208 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188211 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188213 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188216 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188218 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.188236 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188220 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188225 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188230 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188233 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188236 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188247 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188250 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188253 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188255 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188258 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188260 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188263 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188265 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188268 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188270 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188273 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188276 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188278 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188280 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.188723 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188283 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188285 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188288 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188291 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188294 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188297 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188299 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188302 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188304 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188307 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188309 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188313 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188315 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188318 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188320 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188323 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188326 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188329 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188331 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.189216 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188334 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188336 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188339 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188342 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188346 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188348 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188351 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188354 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188356 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188359 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188361 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188363 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188366 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188368 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188371 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188373 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188376 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188379 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188381 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.189675 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188384 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188386 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188388 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188391 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188393 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188399 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188402 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188405 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188407 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188410 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188412 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188415 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188418 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188420 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188423 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188425 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188428 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188431 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188433 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188436 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.190151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188438 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188441 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188443 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188445 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188448 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188451 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188454 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188456 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.188462 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189228 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189235 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189239 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189242 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189244 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189247 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189250 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189252 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189257 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189259 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.190615 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189263 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189268 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189271 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189274 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189277 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189281 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189283 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189286 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189289 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189292 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189294 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189297 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189299 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189302 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189305 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189307 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189310 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189312 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189315 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.191153 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189318 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189320 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189323 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189326 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189328 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189331 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189333 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189336 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189338 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189341 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189343 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189347 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189349 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189352 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189354 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189357 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189359 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189361 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189365 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189367 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.191612 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189370 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189373 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189375 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189378 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189380 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189383 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189385 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189388 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189390 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189392 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189395 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189398 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189400 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189402 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189405 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189407 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189410 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189412 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189415 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189417 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.192126 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189420 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189422 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189424 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189429 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189432 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189434 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189436 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189439 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189441 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189443 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189446 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189448 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189451 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189454 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189456 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189458 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189461 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189463 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189466 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.192610 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189468 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189470 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189473 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189475 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189478 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189480 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189483 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189485 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189487 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189490 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189493 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189495 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189499 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189502 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189504 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189506 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189511 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.189514 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190114 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190120 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.193123 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190124 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190126 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190129 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190132 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190135 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190138 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190140 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190143 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190145 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190148 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190150 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190152 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190155 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190157 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190160 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190163 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190166 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190168 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190172 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.193604 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190175 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190179 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190182 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190191 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190194 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190196 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190199 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190201 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190204 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190209 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190212 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190214 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190216 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190219 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190221 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190224 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190226 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190229 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190231 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190234 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.194076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190236 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190238 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190241 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190244 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190246 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190248 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190251 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190253 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190256 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190258 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190261 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190263 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190267 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190271 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190274 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190277 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190280 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190283 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190285 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.194575 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190287 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190290 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190294 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190296 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190299 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190301 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190304 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190306 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190309 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190311 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190314 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190316 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190319 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190321 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190323 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190326 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190329 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190332 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190334 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.195037 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190337 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190339 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190341 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190344 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190346 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190349 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190351 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190354 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190357 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190359 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190361 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190365 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190367 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190369 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190372 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190375 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190378 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190380 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190382 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.195501 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190385 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190388 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190390 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190393 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190396 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190398 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190401 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.190403 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190501 5548 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190510 5548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190517 5548 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190521 5548 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190527 5548 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190533 5548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190537 5548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190541 5548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190544 5548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190547 5548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190550 5548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190553 5548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190556 5548 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190560 5548 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190562 5548 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:43:29.195955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190565 5548 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190568 5548 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190651 5548 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190662 5548 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190665 5548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190669 5548 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190672 5548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190675 5548 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190680 5548 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190682 5548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190685 5548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190689 5548 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190691 5548 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190694 5548 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190697 5548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190700 5548 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190703 5548 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190707 5548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190714 5548 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190718 5548 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190722 5548 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190725 5548 flags.go:64] FLAG: --enable-server="true" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190727 5548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190731 5548 flags.go:64] FLAG: --event-burst="100" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190735 5548 flags.go:64] FLAG: --event-qps="50" Oct 30 00:43:29.196508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190737 5548 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190740 5548 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190743 5548 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190747 5548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190752 5548 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190756 5548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190760 5548 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190763 5548 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190766 5548 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190769 5548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190772 5548 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190775 5548 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190778 5548 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190781 5548 flags.go:64] FLAG: --feature-gates="" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190785 5548 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190788 5548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190791 5548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190794 5548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190797 5548 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190801 5548 flags.go:64] FLAG: --help="false" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190803 5548 flags.go:64] FLAG: --hostname-override="ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190806 5548 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190809 5548 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:43:29.197115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190812 5548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190815 5548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190820 5548 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190823 5548 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190825 5548 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190828 5548 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190831 5548 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190834 5548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190837 5548 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190839 5548 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190842 5548 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190845 5548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190848 5548 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190851 5548 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190853 5548 flags.go:64] FLAG: --lock-file="" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190875 5548 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190879 5548 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190881 5548 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190887 5548 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190890 5548 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190894 5548 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190897 5548 flags.go:64] FLAG: --logging-format="text" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190899 5548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190902 5548 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:43:29.197940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190905 5548 flags.go:64] FLAG: --manifest-url="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190908 5548 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190912 5548 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190915 5548 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190919 5548 flags.go:64] FLAG: --max-pods="110" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190922 5548 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190925 5548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190927 5548 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190930 5548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190934 5548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190938 5548 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190941 5548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190948 5548 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190951 5548 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190954 5548 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190957 5548 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190960 5548 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190962 5548 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190965 5548 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190968 5548 flags.go:64] FLAG: --port="10250" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190971 5548 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190974 5548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-009549338b3f1a679" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190977 5548 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190980 5548 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190983 5548 flags.go:64] FLAG: --register-node="true" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190986 5548 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:43:29.198792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190990 5548 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190993 5548 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190996 5548 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.190999 5548 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191003 5548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191006 5548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191009 5548 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191012 5548 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191014 5548 flags.go:64] FLAG: --runonce="false" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191017 5548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191020 5548 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191023 5548 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191026 5548 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191028 5548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191031 5548 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191034 5548 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191039 5548 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191042 5548 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191044 5548 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191047 5548 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191050 5548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191053 5548 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191056 5548 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191058 5548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191063 5548 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:43:29.199538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191066 5548 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191069 5548 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191073 5548 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191076 5548 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191079 5548 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191081 5548 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191086 5548 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191089 5548 flags.go:64] FLAG: --v="2" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191099 5548 flags.go:64] FLAG: --version="false" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191103 5548 flags.go:64] FLAG: --vmodule="" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191107 5548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191111 5548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191247 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191251 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191254 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191257 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191259 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191262 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191265 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191267 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191269 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191272 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.200155 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191275 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191279 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191283 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191286 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191289 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191291 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191294 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191296 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191298 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191301 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191303 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191306 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191308 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191310 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191312 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191315 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191319 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191321 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191324 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.200912 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191327 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191329 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191332 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191335 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191338 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191340 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191342 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191345 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191348 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191350 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191353 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191355 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191357 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191360 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191362 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191364 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191367 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191369 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191372 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191375 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.201511 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191378 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191381 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191383 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191386 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191388 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191390 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191393 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191395 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191398 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191401 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191403 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191406 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191408 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191411 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191413 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191415 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191418 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191421 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191423 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191426 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.202060 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191428 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191431 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191433 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191436 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191438 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191441 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191446 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191450 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191453 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191456 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191458 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191461 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191463 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191466 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191468 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191471 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191473 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191476 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191478 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191480 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.202570 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191483 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191487 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191489 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191492 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191494 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191497 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191499 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191503 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191507 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191511 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191514 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191516 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191519 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191521 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191524 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191526 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.203182 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191529 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.191555 5548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191588 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191592 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191595 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191598 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191600 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191603 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191607 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191609 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191612 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191615 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191618 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191620 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191623 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191626 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191629 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.203600 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191632 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191635 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191637 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191640 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191643 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191645 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191648 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191650 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191653 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191656 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191658 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191661 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191663 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191666 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191668 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191672 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191676 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191679 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191681 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.204086 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191685 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191687 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191689 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191692 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191695 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191698 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191700 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191703 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191705 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191708 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191710 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191713 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191715 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191719 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191721 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191724 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191726 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191730 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191734 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.204690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191738 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191741 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191745 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191749 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191753 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191757 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191759 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191762 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191764 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191766 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191769 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191771 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191773 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191776 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191779 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191781 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191784 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191787 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191790 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.205180 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191792 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191795 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191797 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191800 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191802 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191805 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191807 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191811 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191813 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191816 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191818 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191820 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191823 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191826 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191828 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191831 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191833 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191836 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191839 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.205627 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191841 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191844 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191846 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191849 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191852 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191872 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191875 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191877 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191880 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191882 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191885 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191890 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191892 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191895 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.191897 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.195517 5548 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.195530 5548 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.195548 5548 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.195553 5548 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195608 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.206101 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195615 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195620 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195624 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195629 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195633 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195637 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195641 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195645 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195649 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195654 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195660 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195667 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195671 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195674 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195678 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195682 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195685 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195688 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195691 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195693 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.206572 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195696 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195698 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195701 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195704 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195706 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195709 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195711 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195714 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195717 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195719 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195722 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195724 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195726 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195728 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195731 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195733 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195736 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195738 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195740 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195743 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.207067 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195745 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195748 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195750 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195752 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195756 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195759 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195761 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195764 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195766 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195769 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195771 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195774 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195776 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195778 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195781 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195784 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195786 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195789 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195791 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.207566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195793 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195796 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195798 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195800 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195803 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195805 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195807 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195810 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195813 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195815 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195817 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195820 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195822 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195824 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195827 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195829 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195832 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195834 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195837 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.208132 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195840 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195842 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195845 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195847 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195850 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195852 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195855 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195873 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195877 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195879 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195882 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195884 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195887 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195889 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195892 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195895 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195897 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195899 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195902 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.208590 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195904 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195907 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195909 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195911 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195914 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195916 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195919 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.195922 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.195949 5548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196083 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196089 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196093 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196095 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196099 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196101 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196104 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.209075 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196106 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196109 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196111 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196114 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196116 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196118 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196121 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196124 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196126 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196129 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196131 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196133 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196136 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196138 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196141 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196143 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196145 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196148 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196150 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196153 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.209455 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196155 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196158 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196160 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196163 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196165 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196174 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196178 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196180 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196183 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196185 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196188 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196191 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196193 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196196 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196198 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196200 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196203 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196205 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196208 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196210 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.209942 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196213 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196215 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196217 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196220 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196222 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196225 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196227 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196230 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196232 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196234 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196237 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196239 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196244 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196247 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196249 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196252 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196254 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196257 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196259 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.210411 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196262 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196264 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196267 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196269 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196272 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196275 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196278 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196281 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196285 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196287 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196290 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196292 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196294 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196297 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196299 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196302 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196304 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196307 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196309 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.210872 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196312 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196314 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196316 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196318 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196321 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196323 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196327 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196329 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196332 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196334 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196338 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196342 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196345 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196348 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196350 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196353 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196355 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196358 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196361 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196364 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.211321 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196366 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.196391 5548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196420 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196424 5548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196427 5548 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196429 5548 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196432 5548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196434 5548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196437 5548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196440 5548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196444 5548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196447 5548 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196450 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196453 5548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196455 5548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196457 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:29.211796 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196460 5548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196462 5548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196465 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196468 5548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196471 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196473 5548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196476 5548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196478 5548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196482 5548 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196485 5548 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196487 5548 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196489 5548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196493 5548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196495 5548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196498 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196501 5548 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196505 5548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196508 5548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196511 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:29.212208 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196513 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196516 5548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196519 5548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196521 5548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196524 5548 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196527 5548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196529 5548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196531 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196534 5548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196536 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196539 5548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196541 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196544 5548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196546 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196548 5548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196551 5548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196554 5548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196556 5548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196559 5548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196562 5548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:29.212690 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196564 5548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196568 5548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196571 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196573 5548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196576 5548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196578 5548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196581 5548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196583 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196586 5548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196588 5548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196591 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196593 5548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196595 5548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196598 5548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196600 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196602 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196605 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196607 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196610 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:29.213445 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196612 5548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196614 5548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196617 5548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196620 5548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196622 5548 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196624 5548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196627 5548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196629 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196631 5548 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196634 5548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196637 5548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196639 5548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196641 5548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196644 5548 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196647 5548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196650 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196652 5548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196655 5548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196657 5548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196659 5548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:29.214131 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196662 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196664 5548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196666 5548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196670 5548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196672 5548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196675 5548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196678 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196680 5548 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196682 5548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196685 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196687 5548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196689 5548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196692 5548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:43:29.196694 5548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.196740 5548 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.198645 5548 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.198692 5548 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.198886 5548 server.go:1014] "Starting client certificate rotation" Oct 30 00:43:29.214674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.199024 5548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.199069 5548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:37:19 +0000 UTC" deadline="2028-05-10 23:30:24.531461651 +0000 UTC" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.199098 5548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13414h46m55.332365302s" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.199306 5548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.200374 5548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.201643 5548 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.203583 5548 log.go:25] "Validated CRI v1 image API" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.204280 5548 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:43:29.215209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.206286 5548 fs.go:135] Filesystem UUIDs: map[538e9656-c3e4-4fef-bc50-c7ef99eacda6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ed33f9d1-7352-4cee-bb52-8c6d18887643:/dev/nvme0n1p3] Oct 30 00:43:29.215465 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.206299 5548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d738ef0005d5bace2e856b164fbee0f1fa92fd4623aefd675a46ccd344b957e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d738ef0005d5bace2e856b164fbee0f1fa92fd4623aefd675a46ccd344b957e/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/986278046d4a44c9ab26f0be8ef21e3fc1accf36c41655f5904a1ff233fd4d17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/986278046d4a44c9ab26f0be8ef21e3fc1accf36c41655f5904a1ff233fd4d17/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/043e9d07-b27e-4378-b055-c4d5bdcc29e3/volumes/kubernetes.io~projected/kube-api-access-khz2k:{mountpoint:/var/lib/kubelet/pods/043e9d07-b27e-4378-b055-c4d5bdcc29e3/volumes/kubernetes.io~projected/kube-api-access-khz2k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~projected/kube-api-access-j8rqg:{mountpoint:/var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~projected/kube-api-access-j8rqg major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab30191-af9f-4578-ae26-c57232aa5111/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1ab30191-af9f-4578-ae26-c57232aa5111/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368c928a-a45a-4446-942b-39bd53f20a04/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/368c928a-a45a-4446-942b-39bd53f20a04/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~projected/kube-api-access-qbsk6:{mountpoint:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~projected/kube-api-access-qbsk6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49dd9e0f-977b-454c-8599-f6357642c0b6/volumes/kubernetes.io~projected/kube-api-access-skmnj:{mountpoint:/var/lib/kubelet/pods/49dd9e0f-977b-454c-8599-f6357642c0b6/volumes/kubernetes.io~projected/kube-api-access-skmnj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b428a8-331e-43e7-bfc1-fe7e0e6732fe/volumes/kubernetes.io~projected/kube-api-access-llh49:{mountpoint:/var/lib/kubelet/pods/54b428a8-331e-43e7-bfc1-fe7e0e6732fe/volumes/kubernetes.io~projected/kube-api-access-llh49 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~projected/kube-api-access-q24vm:{mountpoint:/var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~projected/kube-api-access-q24vm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9/volumes/kubernetes.io~projected/kube-api-access-2njcm:{mountpoint:/var/lib/kubelet/pods/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9/volumes/kubernetes.io~projected/kube-api-access-2njcm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/alertmanager/9 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/kube-api-access-cvzfw:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/kube-api-access-cvzfw major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/tls-assets major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/config-volume major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/web-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~projected/kube-api-access-8r5mr:{mountpoint:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~projected/kube-api-access-8r5mr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~projected/kube-api-access-m94j8:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~projected/kube-api-access-m94j8 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2abc6d-480c-4b36-a525-ab1e04bf57c0/volumes/kubernetes.io~projected/kube-api-access-wjwl4:{mountpoint:/var/lib/kubelet/pods/7f2abc6d-480c-4b36-a525-ab1e04bf57c0/volumes/kubernetes.io~projected/kube-api-access-wjwl4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~projected/kube-api-access-24fhd:{mountpoint:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~projected/kube-api-access-24fhd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~projected/kube-api-access-jndl8:{mountpoint:/var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~projected/kube-api-access-jndl8 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~projected/kube-api-access-w6zbd:{mountpoint:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~projected/kube-api-access-w6zbd major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~projected/kube-api-access-tsqhg:{mountpoint:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~projected/kube-api-access-tsqhg major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~projected/kube-api-access-58lbv:{mountpoint:/var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~projected/kube-api-access-58lbv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~secret/metrics-certs major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e469533b-597a-4cf2-afd5-ad2fa146a978/volumes/kubernetes.io~projected/kube-api-access-cf4w7:{mountpoint:/var/lib/kubelet/pods/e469533b-597a-4cf2-afd5-ad2fa146a978/volumes/kubernetes.io~projected/kube-api-access-cf4w7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~projected/kube-api-access-jmdk5:{mountpoint:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~projected/kube-api-access-jmdk5 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed92d686-6ba7-49de-b947-7178cb50b3c0/volumes/kubernetes.io~projected/kube-api-access-vtjfh:{mountpoint:/var/lib/kubelet/pods/ed92d686-6ba7-49de-b947-7178cb50b3c0/volumes/kubernetes.io~projected/kube-api-access-vtjfh major:0 minor:416 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fb69f30d3853a97b9227c8cb5a4a54f055b2a63f32c3594ddc1ac5ceec29165d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5fcc36083fe058b5c7631bad2dced2943c41a0d45b4dd5c1f24c2649bbb4365e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f105ca444584d79fb45c1cb704952be231331f79e0620b2495cfcd308746f829/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1e68bebdfd55fb3398c8fe7bc88baa56d6a166d4327810580dca2b064f08f20c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1c93b8595c1b8b60f7a69346a365eaad8fe629547c5d1a7c244420fa3c8ef5c2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/013133bfff1dea8ec8776820015872b9003c345c8f11e5e6e239ca17e51a5592/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/65135b3313f4f4b9cca82ef8d23c4ede01efbe1ed39dc49efcd1d609e1774993/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/baba38aba107eeedf1c9b43c24659aec3439e887424887519b0114253c2fb283/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f42c62d58a10ea5bd07edd566fcd8a9943d12810429dfe5eccf59c8fd433ac23/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/c8e4398ba95c4d23cef902f1b9a1614b17ccc2148c3598ffd11d9e35ead7a21a/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7f39c69efb95922bdbeb1ab1dc22caf3ad7da42a97b7bc521b7600a9a17ba776/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/a696b6181a8fc1306ae4c34f411b783b39387b31a71ee9d6f83b8298403208cc/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/b214b051e4bb598e9016bc428c1f92c7bb53da5db2681ae661300937feaed78d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cc6f6425825793a04529f331e1d1bdb680fb39870faa6643724f8a3a56f5a0e1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/c88a8705fe7b75cf56b24304b0705fea9dade2a5aaa715e796435e94661405d7/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1fb257ea8b29221c4f7fcd7f57428f53a9fb6b8507b5c007282d40f139496da3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/83f33a8fb7bed7e4e2ea884354d9b59701a76876510d3b778902087ccb6b5893/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0638ed3a5bdab3029370f074468401fddbb8b079da4138e758d49ac7a76d9cb7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/74b4f81f5a88653f2f6a03ecefd02fef9e82d051af32bf0cffadf060c64b024f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2c9204d89fbc2cf4b5af34d4fa549bf1321d5d198092e28e977e068201f8822d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c478d605522d8289ae06bb159fe9bbd37d69f1648cb40b6172984b92be1cb3f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f9bdff12c27cb286cd03c0dbabaaeb590ffdbf7c9fb88145ba3cc497ffa3ff99/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/78b52972ce05680251f9798a48b6f3cc2888554b9f8ad4f3b1760bab7a45ba31/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/b20bf08611ff65c8444467c29bb948664e2798773b900f35fb535753b62461e0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/a04fa16a345ba067af15dab42483e103f06655e1be9b2d3401ceff92d49fa918/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8973442e1bb7a4ad777c08d3a044f7b0787b90b4aa1470f7eb80a634d6ecfd32/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/926530f6038ede22f6ca87d4a307d81d33545ef9629ba6cf4ca4c0783e6b8d20/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/02054cead614f0491638a22b9aca024a56b55bd8f91773d1451fde84032efecc/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/72e974052624efd054c99b88bbef973df03d1520de05801e45c1617541c237d1/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/4c49c587655411db6b992fc100e94b10a2a5ee59c2a7d4ece908867dc0fcf6db/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/cd7899113562f0de4885a9d0b6c2bacadebf12b9a7f333740e8592dfd9da605b/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/3a78a2f07e700d3f5486c55adef493276e8cffe633004e1069d194340e4e6f9d/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0f9245e26eaa4847fcff817391565fe2f51b67af94cd5cf2a28f1851db9c6050/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1faab137c10b15d6eb93fea22d2debc9023305e594c997b0d6aa04df15c12ca8/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/7d299d4a1b4b4fee5ce2d6fad5232752f28c1c9f0afb07a505c8511c1cd05107/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/d62ebb01ada6a32daa069bdff80ae38b76e984c69f8695042e362500fd586695/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/dc106f630a5fa5050fc6f3dc179edc4cf10ec282b36073d11c000ee1e6a913c4/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/e8ecfe64a91f03b3c7bb4e7fe7861cd7dc7b90bd0dfb001a390fcf60d978bc5d/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/b13ec68cb095c51ea78f55f22c335f272b44be595fce65b7efc4c261745b974b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/0a924ba9814c75bd43afc2d1cbd7ddd4febc5cce9593fb818a363e3ce947eec0/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a2fe6397d5b1980da2c35c6b6ca153841d6acf8fb3497b7d4f10a00e82c51764/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/b182f63652fbd6c0fbf79b34cf6dd5327b77e0a5d205adaac4ccd582b81f9ac8/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/4fe1d2cf9a8929520df2defddcf17550d60eb5de8f95f22c3d50029344a662d7/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/77cf27d60b07257981c0c7dbf4780dc978847edc7b99d7d4a5a1c423841f8744/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/adb428cc457b16845e885f275596fe144a4c9ff84b1af27be857051db31569df/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/6e1a93535dc592c05301e6a8afb2eed1cadd9b5e238128cd0d9660c86ef658e2/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/1b53cc81a3a89f16d41c52f3f6b1c135e68e4fe48ac791d29cebfe5fee056a3b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/738173a7c703bd217afc118aa0dbbfd57ad949cc225dddb8d427d59d45c812dc/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/722e8a2bfeb8456dd47c328dca7986255b0dcffcc8ad86cc802ae5fa26e66b3e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/abe97855297e34259edb2c24c22c796fe70e933c354fd854ba84f54b38a1c1c0/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/6aa70e3ba3a1fe5fd8fb74865cee4ec2ee10d7e6766939382c9027f69e655658/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/bb26cf25d79ed900153f70903c9c084630de03c3707193f055967579d1cc16f1/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/d39ab7ebdfab6e6b7ea2f880999b0d1b5bcf9bc17b5305b33a846f9fb265f715/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/a784edf326d8e35c63ba0e894ed52c307fc5eb82a51ab7d35aefc367ce753bde/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/80628463780aad75bcac57ec417e0ce4a3dd7d5b3a598954942983c5aa1db7fd/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/b18d66792d11abb4ba08fed9c9ddb681747ae07de58df2c1f54f2ec44e247767/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/6c274840bf72b6b6b1dccc9c431ec1667971aa9e46430cc5ef66e4d42a9bef36/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/1344ae20bc260156e6178e365837077f129e1e5cc60cc40bea6ac0595c6358b0/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/2640c5fbf161c46609f91ca7185cca15e6950de24c6e86c612118fdb2d07e3d2/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/a9be82e1d9a9d3ad94e6f52f3fa9850b63364ec3865910c5c970011f8c802eb2/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/44025a339ce05d5cfb85c87b8994361806ae89e39b3bfd7a08a6be1c54e31eee/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/0af2b25668a962e99a763d54aa463d6604366707555d9f54d537559fc41c66a9/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/5f7faa5da09854da939628cebd1476719d58cd5be14663ab953bf3947ede8ce9/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/becef753eb7c338aecd5279433171d16874275e3b38025a310a47f729c12f064/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a9a2dd7590ded7c1373847c57cdb5fdb819971f53722266442091d70f5af813f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/94b39e6e91bc6503a05cf7722397a3a2ac54fb5503308f46c1970abc4ca6602e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0cdd99592f6d3e0a7d4d39c09d258c8bace1617561062d39806bc2dc1c2067ba/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c88ed83f219b1a53052764dee11cdcd261880243c49a1bf274b390e391cfef4c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ef9c3a35f0d2591d12ee14e820ca27285cb926d20cb0b2de47c1555a477dd36e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/04df641c49d522f5d2425a951541624533a7258e1d916f56123bd09beafe79bf/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/77855564e4d4fb73e715d01ee4f63a32185ce58a340908db75458d70c6767f33/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8ed2ddda604aaf52b4f15df45c8de6e40ea255e3a0cfed39d523bcaa30a1fc91/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/8e6cdfd0a1895b3fbec6e53637ad69715c0017026fa31af1899ce0e174e52276/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2e88a0073851c5994c6be310358010ed8289765d920e84fb029e5932ad211d3a/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.214576 5548 manager.go:217] Machine: {Timestamp:2026-10-30 00:43:29.213721567 +0000 UTC m=+0.056282153 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3154223 MemoryCapacity:32811999232 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b693d8cdec017eec19fae2806f928 SystemUUID:ec2b693d-8cde-c017-eec1-9fae2806f928 BootID:8e057b75-3051-4f1a-8d8c-4c520ba04a55 Filesystems:[{Device:/var/lib/kubelet/pods/54b428a8-331e-43e7-bfc1-fe7e0e6732fe/volumes/kubernetes.io~projected/kube-api-access-llh49 DeviceMajor:0 DeviceMinor:66 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~projected/kube-api-access-jmdk5 DeviceMajor:0 DeviceMinor:341 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:281 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/49dd9e0f-977b-454c-8599-f6357642c0b6/volumes/kubernetes.io~projected/kube-api-access-skmnj DeviceMajor:0 DeviceMinor:64 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/986278046d4a44c9ab26f0be8ef21e3fc1accf36c41655f5904a1ff233fd4d17/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/043e9d07-b27e-4378-b055-c4d5bdcc29e3/volumes/kubernetes.io~projected/kube-api-access-khz2k DeviceMajor:0 DeviceMinor:60 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:233 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:280 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32aedd675a0828653c2120cd6ac76b7c1ddd439ab7071ff854c27c36bd832152/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9/volumes/kubernetes.io~projected/kube-api-access-2njcm DeviceMajor:0 DeviceMinor:65 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/923ef7a9c82bf1b508407d039f6d3461e4450ce94fb36c6fad2655cbf0a9006a/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:279 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/kube-api-access-cvzfw DeviceMajor:0 DeviceMinor:285 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb09cd18a5efb6f5903a50d593c84583046aa59dbd9a9920bb06b0e7d3f5440e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:332 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~projected/kube-api-access-m94j8 DeviceMajor:0 DeviceMinor:333 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab30191-af9f-4578-ae26-c57232aa5111/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:415 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~projected/kube-api-access-jndl8 DeviceMajor:0 DeviceMinor:204 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~projected/kube-api-access-w6zbd DeviceMajor:0 DeviceMinor:236 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~projected/kube-api-access-24fhd DeviceMajor:0 DeviceMinor:59 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:410 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8704039bb8179a05e740de8a1cf8b657d1e8b84c72afe3c3746fa204bf76cf61/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~projected/kube-api-access-j8rqg DeviceMajor:0 DeviceMinor:203 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:328 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faead933fdd41cf0948067ae1bc0a9fb99f069d26d7224ea30f607a341053015/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1437846a-8b02-48e2-8963-d04d2432ecae/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84a64ab37099d1d411d11861d9704843a8e95fb5e9c587be33c2f6d53d055719/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/989f29d19f794ccd7ed1cf2980638f8fc8b00c132debbd1c30dbf52923cf9c90/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:278 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69811e5e763c1b1f63179244761a32d8bb0cd8d1070997734160a9aaed4ff28f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34bab93ae411c8033346f46b95a0680d1ecb9ad046911b8cbded3dae6466cd76/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7c43657-fe00-4520-9ee8-684223518dec/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16405999616 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:284 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:324 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed92d686-6ba7-49de-b947-7178cb50b3c0/volumes/kubernetes.io~projected/kube-api-access-vtjfh DeviceMajor:0 DeviceMinor:416 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:282 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:273 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/930b2441d8c641b90ceb3a9f38a947ad6d4f5f4329c056bb639289b8a8123498/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:277 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bad174eaf14d19f56ee611178334a03ae912e2af68dccec622ecadaa5396f347/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7754f668ab9b68ee0d814149803aa8b3c3a5f2c82ba0f57649e745e97880a85a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~projected/kube-api-access-q24vm DeviceMajor:0 DeviceMinor:197 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d738ef0005d5bace2e856b164fbee0f1fa92fd4623aefd675a46ccd344b957e/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:273 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:330 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7102eb-69ac-4caf-8039-18466d995097/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58ed2301-70f6-497f-b717-be08bdb686c4/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeb58a8-a0fe-48f2-86a7-0e347a585522/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volumes/kubernetes.io~projected/kube-api-access-8r5mr DeviceMajor:0 DeviceMinor:62 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69fd86b2-89ac-4e07-8dd7-da1cf88785c3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7335392212903d7fd70e90969da4f3740d25b34abaf6e32ca17b064e2e4abe9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a80b6429-b22b-4105-a70a-799faed22645/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/40834bbb-90ae-486f-bbcb-ff665c3f313e/volumes/kubernetes.io~projected/kube-api-access-qbsk6 DeviceMajor:0 DeviceMinor:235 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a666b42d71e072eab6e953a3733c96160ca9aa5e4809010a1ad1701eb5c50d22/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/volumes/kubernetes.io~projected/kube-api-access-tsqhg DeviceMajor:0 DeviceMinor:234 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7df6ae982bedf8fc89be1b7c139aafd4f0ea629d26cae8c612425934392f6a2e/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/368c928a-a45a-4446-942b-39bd53f20a04/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ee4dc6ba2a59aaaffc9173f6879ad9cba470132f9fd8a4c25df1048e024cc75/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:280 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7d248e8-94d3-40dd-baf3-9cfb15053504/volumes/kubernetes.io~projected/kube-api-access-58lbv DeviceMajor:0 DeviceMinor:67 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/e469533b-597a-4cf2-afd5-ad2fa146a978/volumes/kubernetes.io~projected/kube-api-access-cf4w7 DeviceMajor:0 DeviceMinor:61 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2abc6d-480c-4b36-a525-ab1e04bf57c0/volumes/kubernetes.io~projected/kube-api-access-wjwl4 DeviceMajor:0 DeviceMinor:63 Capacity:31633399808 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52ee5058922f933088da339ddbb36c4ada6cbe4f6023943bc8cbe2832758a064/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ee4dc6ba2a59aa MacAddress:8e:e3:4a:3f:cf:4c Speed:10000 Mtu:8901} {Name:4d738ef0005d5ba MacAddress:9a:80:ed:74:4f:66 Speed:10000 Mtu:8901} {Name:69811e5e763c1b1 MacAddress:f6:62:cb:1e:fd:b7 Speed:10000 Mtu:8901} {Name:7df6ae982bedf8f MacAddress:ee:37:9a:59:f3:5c Speed:10000 Mtu:8901} {Name:84a64ab37099d1d MacAddress:ba:49:ef:85:bf:84 Speed:10000 Mtu:8901} {Name:8704039bb8179a0 MacAddress:0e:03:e3:7d:2b:02 Speed:10000 Mtu:8901} {Name:930b2441d8c641b MacAddress:62:7f:10:0c:46:3e Speed:10000 Mtu:8901} {Name:986278046d4a44c MacAddress:76:86:7c:ed:a7:bd Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:7d:3e:45 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:0e:32:9f:0e:d8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:7d:3e:45 Speed:0 Mtu:9001} {Name:f888425ab2606f8 MacAddress:b2:c4:a3:1d:5a:ec Speed:10000 Mtu:8901} {Name:faead933fdd41cf MacAddress:fa:fe:87:cd:e5:53 Speed:10000 Mtu:8901} {Name:fb09cd18a5efb6f MacAddress:02:77:30:dd:14:eb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:32:01:fa:ab:13 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:cb:61:76:93:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32811999232 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215002 5548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215047 5548 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215318 5548 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215337 5548 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-7.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215455 5548 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215462 5548 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215475 5548 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215494 5548 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:43:29.215698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215580 5548 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215730 5548 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215838 5548 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215892 5548 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215908 5548 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215918 5548 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215927 5548 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:43:29.216260 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.215935 5548 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:43:29.216881 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.216851 5548 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:43:29.216915 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.216885 5548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:43:29.217390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.217371 5548 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:43:29.219939 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.219924 5548 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:43:29.219939 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.219937 5548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:43:29.220051 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220041 5548 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:43:29.220248 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220236 5548 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:43:29.220280 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220266 5548 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:43:29.220367 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220358 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220373 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220379 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220385 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220391 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220397 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220404 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:43:29.220413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220413 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220421 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220426 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220442 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220451 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220474 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:43:29.220632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.220484 5548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:43:29.223811 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.223799 5548 server.go:1263] "Started kubelet" Oct 30 00:43:29.224392 ip-10-0-143-7 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:43:29.224611 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.224545 5548 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:43:29.224706 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.224629 5548 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:43:29.224706 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.224575 5548 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:43:29.225558 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.225537 5548 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:43:29.226294 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.226257 5548 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.228531 5548 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.228808 5548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.232138 5548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-7.ec2.internal\" not found" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.232190 5548 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.232199 5548 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:43:29.232905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.232575 5548 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:43:29.233948 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.233905 5548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:20 +0000 UTC" deadline="2028-04-27 14:51:56.232333712 +0000 UTC" Oct 30 00:43:29.234038 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.233950 5548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13094h8m26.998387704s" Oct 30 00:43:29.236349 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236325 5548 factory.go:55] Registering systemd factory Oct 30 00:43:29.236349 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236347 5548 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236615 5548 factory.go:153] Registering CRI-O factory Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236635 5548 factory.go:223] Registration of the crio container factory successfully Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236689 5548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236715 5548 factory.go:103] Registering Raw factory Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.236728 5548 manager.go:1199] Started watching for new ooms in manager Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237215 5548 manager.go:319] Starting recovery of all containers Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237318 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237366 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ed2301-70f6-497f-b717-be08bdb686c4" volumeName="kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237380 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6266e78f-7ca5-43bf-b3e6-6206fc45c4c9" volumeName="kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237392 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" volumeName="kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237415 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237426 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7d248e8-94d3-40dd-baf3-9cfb15053504" volumeName="kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237438 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="043e9d07-b27e-4378-b055-c4d5bdcc29e3" volumeName="kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237459 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237473 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" volumeName="kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237484 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237504 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237516 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e469533b-597a-4cf2-afd5-ad2fa146a978" volumeName="kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237528 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b428a8-331e-43e7-bfc1-fe7e0e6732fe" volumeName="kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237561 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237576 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c43657-fe00-4520-9ee8-684223518dec" volumeName="kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237587 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c43657-fe00-4520-9ee8-684223518dec" volumeName="kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237638 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c43657-fe00-4520-9ee8-684223518dec" volumeName="kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237649 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237812 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e469533b-597a-4cf2-afd5-ad2fa146a978" volumeName="kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237845 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237874 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="043e9d07-b27e-4378-b055-c4d5bdcc29e3" volumeName="kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237886 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237921 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a80b6429-b22b-4105-a70a-799faed22645" volumeName="kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237953 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dd9e0f-977b-454c-8599-f6357642c0b6" volumeName="kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.237965 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238358 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" volumeName="kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238386 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab30191-af9f-4578-ae26-c57232aa5111" volumeName="kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238399 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238438 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238464 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1437846a-8b02-48e2-8963-d04d2432ecae" volumeName="kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238596 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c928a-a45a-4446-942b-39bd53f20a04" volumeName="kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238627 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238648 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238661 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238688 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f159ccf-7e4f-45d2-86e5-1b40e51df4b3" volumeName="kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238699 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7c43657-fe00-4520-9ee8-684223518dec" volumeName="kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238711 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238772 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b428a8-331e-43e7-bfc1-fe7e0e6732fe" volumeName="kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238784 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" volumeName="kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238795 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2abc6d-480c-4b36-a525-ab1e04bf57c0" volumeName="kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238819 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238831 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238877 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238901 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238930 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368c928a-a45a-4446-942b-39bd53f20a04" volumeName="kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238941 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ed2301-70f6-497f-b717-be08bdb686c4" volumeName="kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.238999 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239019 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7d248e8-94d3-40dd-baf3-9cfb15053504" volumeName="kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239038 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239069 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ed2301-70f6-497f-b717-be08bdb686c4" volumeName="kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239100 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239116 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239156 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a80b6429-b22b-4105-a70a-799faed22645" volumeName="kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239169 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239196 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dd9e0f-977b-454c-8599-f6357642c0b6" volumeName="kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239227 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239247 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239258 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239307 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239318 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ed2301-70f6-497f-b717-be08bdb686c4" volumeName="kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239329 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239349 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a80b6429-b22b-4105-a70a-799faed22645" volumeName="kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239360 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239372 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b428a8-331e-43e7-bfc1-fe7e0e6732fe" volumeName="kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239404 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239416 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66681562-5027-4a5d-9e34-762f99b60322" volumeName="kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239436 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239459 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239490 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239510 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f159ccf-7e4f-45d2-86e5-1b40e51df4b3" volumeName="kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239531 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1437846a-8b02-48e2-8963-d04d2432ecae" volumeName="kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239542 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dd9e0f-977b-454c-8599-f6357642c0b6" volumeName="kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239552 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49dd9e0f-977b-454c-8599-f6357642c0b6" volumeName="kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239573 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6266e78f-7ca5-43bf-b3e6-6206fc45c4c9" volumeName="kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239625 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeb58a8-a0fe-48f2-86a7-0e347a585522" volumeName="kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239639 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a80b6429-b22b-4105-a70a-799faed22645" volumeName="kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239663 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239675 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bbb-90ae-486f-bbcb-ff665c3f313e" volumeName="kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239687 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" volumeName="kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239709 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f159ccf-7e4f-45d2-86e5-1b40e51df4b3" volumeName="kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239720 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" volumeName="kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239733 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7102eb-69ac-4caf-8039-18466d995097" volumeName="kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239771 5548 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed92d686-6ba7-49de-b947-7178cb50b3c0" volumeName="kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh" seLinuxMountContext="" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239784 5548 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:43:29.241287 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.239792 5548 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:43:29.254841 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.254804 5548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:43:29.256484 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.256463 5548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:43:29.256484 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.256482 5548 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:43:29.256615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.256502 5548 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:43:29.256615 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.256545 5548 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:43:29.266711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.266659 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:43:29.266928 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.266909 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:43:29.267636 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.267592 5548 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:29.268439 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.268415 5548 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:29.268629 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.268610 5548 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:43:29.288735 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.288713 5548 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:29.288923 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.288908 5548 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:29.292565 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.292536 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" exitCode=0 Oct 30 00:43:29.304829 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304805 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="3b2a19b30c3ab9f76f0eb4d888fd2219a10599e0121cfe50a47f42f550141e16" exitCode=0 Oct 30 00:43:29.304829 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304828 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="49b71e7c5c2cf1e568ebd213830dd3726cc2510fc854dd733050b56a8d1a64b3" exitCode=0 Oct 30 00:43:29.304962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304834 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="d2fe78c01d80250f724890c147f4fe84c8dbc67cdfefaac80f990dedcefda717" exitCode=0 Oct 30 00:43:29.304962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304839 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="c65be92ffaffff89582e776efa64e461f46f5a71ae0b22a6a02710afcbc10626" exitCode=0 Oct 30 00:43:29.304962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304844 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="671544b72508f6b995de303c3137458b8befc67078ff5d42f64d236a55f95f6b" exitCode=0 Oct 30 00:43:29.304962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.304850 5548 generic.go:356] "Generic (PLEG): container finished" podID="49dd9e0f-977b-454c-8599-f6357642c0b6" containerID="f3924cf0f094da073906d53918c4d49de555f64206c60f001b3a658d2c9d3a94" exitCode=0 Oct 30 00:43:29.309533 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.309505 5548 generic.go:356] "Generic (PLEG): container finished" podID="aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d" containerID="cf47c7cb6af786abc4cd4be7c11aac32d8e2250b055bdff1c7574a9ad08b158d" exitCode=0 Oct 30 00:43:29.317644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.317628 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:43:29.318022 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.318000 5548 generic.go:356] "Generic (PLEG): container finished" podID="69fd86b2-89ac-4e07-8dd7-da1cf88785c3" containerID="9a330014799fd7213a2e5ecbd8311ca93c8c6f953c5aada935e123a222b2d6e7" exitCode=1 Oct 30 00:43:29.321097 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.321071 5548 generic.go:356] "Generic (PLEG): container finished" podID="767c439e26e5517d9df3fdb3ccab0ad7" containerID="0e35f58ccf5ba45b2940361acd67815919eea34db04e6beccaddc0cfbb7cb3df" exitCode=0 Oct 30 00:43:29.347989 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.347940 5548 manager.go:324] Recovery completed Oct 30 00:43:29.356653 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.356626 5548 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 00:43:29.371052 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371034 5548 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:43:29.371052 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371048 5548 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:43:29.371179 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371065 5548 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:43:29.371226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371196 5548 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:43:29.371226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371208 5548 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:43:29.371226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371224 5548 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:43:29.371337 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371230 5548 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:43:29.371337 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371235 5548 policy_none.go:50] "Start" Oct 30 00:43:29.371337 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371244 5548 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:43:29.371337 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371253 5548 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:43:29.371437 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371426 5548 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:43:29.371437 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371433 5548 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:43:29.371491 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.371439 5548 policy_none.go:44] "Start" Oct 30 00:43:29.375920 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.375891 5548 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:43:29.378492 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.378479 5548 manager.go:341] "Starting Device Plugin manager" Oct 30 00:43:29.379089 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.378505 5548 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:43:29.379147 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379093 5548 server.go:87] "Starting device plugin registration server" Oct 30 00:43:29.379357 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379347 5548 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:43:29.379387 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379359 5548 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:43:29.379456 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379443 5548 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:43:29.379574 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379536 5548 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:43:29.379674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379594 5548 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:43:29.379674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379603 5548 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:43:29.379761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.379645 5548 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:43:29.379570815Z","UUID":"2f26d2d1-90ee-4ed0-a7b1-45f351853dce","Handler":null,"Name":"","Endpoint":""} Oct 30 00:43:29.381067 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.381035 5548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:43:29.381067 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.381063 5548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:43:29.381209 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.381157 5548 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:43:29.479609 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.479574 5548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:43:29.481795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.481776 5548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:43:29.481918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.481812 5548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:43:29.481918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.481826 5548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-7.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:43:29.481918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.481905 5548 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.509737 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.509711 5548 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.509835 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.509791 5548 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.556928 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.556851 5548 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal"] Oct 30 00:43:29.557451 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557419 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" event={"ID":"09bade25a7c8ebe0a0dcc7613db38300","Type":"ContainerStarted","Data":"c26bdc560d2f63cc1c604eb06cd502f31cfc42d6ca6de980dc0eedd82d324f10"} Oct 30 00:43:29.557510 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557464 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" event={"ID":"09bade25a7c8ebe0a0dcc7613db38300","Type":"ContainerStarted","Data":"78a51a1f1e35d673e106350874d83d17b0f34e7107b9d6a5ff48ae78fdbf527f"} Oct 30 00:43:29.557510 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557494 5548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.557632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557562 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerStarted","Data":"46e1954b54de11635dd1a4269d45dc6f95dd1cfe859a8d92a354bf7594436ce1"} Oct 30 00:43:29.557632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557577 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerDied","Data":"0e35f58ccf5ba45b2940361acd67815919eea34db04e6beccaddc0cfbb7cb3df"} Oct 30 00:43:29.557632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557589 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" event={"ID":"767c439e26e5517d9df3fdb3ccab0ad7","Type":"ContainerStarted","Data":"7909b3b9c6d96c5c9ce637ea15f08d95c55362ddbd70dc1e3d16ca6d44687aad"} Oct 30 00:43:29.557779 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.557721 5548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.577991 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.577969 5548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:29.578100 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.578016 5548 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.578365 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.578351 5548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:29.578400 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:29.578381 5548 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.670781 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.670753 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.670894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.670800 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.670894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.670821 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771355 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771311 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771355 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771361 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771489 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771421 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09bade25a7c8ebe0a0dcc7613db38300-config\") pod \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" (UID: \"09bade25a7c8ebe0a0dcc7613db38300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771489 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771440 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771489 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771421 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:29.771630 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:29.771500 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/767c439e26e5517d9df3fdb3ccab0ad7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" (UID: \"767c439e26e5517d9df3fdb3ccab0ad7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:30.216476 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.216443 5548 apiserver.go:52] "Watching apiserver" Oct 30 00:43:30.258689 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.258655 5548 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:43:30.260033 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.259988 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz","openshift-multus/multus-g7tv9","openshift-monitoring/node-exporter-8mh62","openshift-network-operator/iptables-alerter-mst6g","kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm","openshift-cluster-node-tuning-operator/tuned-2gkzk","openshift-multus/multus-additional-cni-plugins-k98jz","openshift-dns/dns-default-8n2dz","openshift-image-registry/node-ca-kp9nh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r","openshift-insights/insights-runtime-extractor-7kjw6","kube-system/global-pull-secret-syncer-ncxvc","openshift-dns/node-resolver-6qb5x","openshift-multus/network-metrics-daemon-mrgqt","openshift-monitoring/metrics-server-5f9c87f897-7zng6","openshift-network-diagnostics/network-check-target-x9kzn","openshift-ovn-kubernetes/ovnkube-node-tcd47","openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf","kube-system/konnectivity-agent-ts2gs","openshift-ingress-canary/ingress-canary-gz2md"] Oct 30 00:43:30.275838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.275818 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.276015 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.275999 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:43:30.276124 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276106 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.276203 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276111 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.276240 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276220 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:30.276240 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276226 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.276316 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276286 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\"" Oct 30 00:43:30.276398 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276381 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\"" Oct 30 00:43:30.276452 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276395 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:30.276452 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276442 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:43:30.276779 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276748 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.276899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276792 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:43:30.276899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276818 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:43:30.276899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276875 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.276899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276821 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\"" Oct 30 00:43:30.276899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276898 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276902 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276795 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276991 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.276993 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277030 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277042 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277046 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277043 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277086 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\"" Oct 30 00:43:30.277139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.277112 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:43:30.285180 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285165 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:43:30.285259 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285235 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.285432 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285419 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.285432 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285429 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.285497 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285463 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.285497 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285487 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:43:30.285641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285630 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:43:30.285752 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285740 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 00:43:30.285945 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285929 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:30.285945 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285929 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.286047 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.285965 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:30.287843 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287813 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:43:30.287843 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287836 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287851 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287905 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287906 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287935 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287936 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287954 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287958 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.287992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.287964 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.288338 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288050 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.288338 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288141 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:43:30.288338 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288144 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:43:30.288338 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288227 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:43:30.288338 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288269 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 00:43:30.288531 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288414 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.288807 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288778 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:43:30.288807 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288801 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:43:30.288807 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288811 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288833 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288873 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288892 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288895 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288903 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:43:30.289023 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.288904 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n682p\"" Oct 30 00:43:30.289272 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289052 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbt7f\"" Oct 30 00:43:30.289272 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289063 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:43:30.289469 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289450 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\"" Oct 30 00:43:30.289546 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289484 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:43:30.289546 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289491 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\"" Oct 30 00:43:30.289993 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.289973 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:43:30.290087 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290007 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 00:43:30.290087 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290045 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:43:30.290332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290312 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:30.290446 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290425 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\"" Oct 30 00:43:30.290680 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290665 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 00:43:30.290751 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290721 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:43:30.290905 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.290891 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 00:43:30.291031 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291018 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6vzmx\"" Oct 30 00:43:30.291090 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291080 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\"" Oct 30 00:43:30.291161 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291149 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\"" Oct 30 00:43:30.291215 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291201 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\"" Oct 30 00:43:30.291319 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291307 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 00:43:30.291387 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291381 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c35a6pei4g2kp\"" Oct 30 00:43:30.291430 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291423 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 00:43:30.291472 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291454 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m82nk\"" Oct 30 00:43:30.291528 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291516 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 00:43:30.291607 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291587 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9p42h\"" Oct 30 00:43:30.291655 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291624 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-sicahncopf47\"" Oct 30 00:43:30.291716 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291687 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b4t5b\"" Oct 30 00:43:30.291759 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291736 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 00:43:30.291803 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291795 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:43:30.291912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.291899 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:43:30.295187 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.295172 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\"" Oct 30 00:43:30.298126 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.298110 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:43:30.324263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.324239 5548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:30.324366 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.324352 5548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:30.334406 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.334391 5548 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:43:30.342319 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.342306 5548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:30.342361 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:30.342341 5548 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-7.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-7.ec2.internal" Oct 30 00:43:30.342649 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.342637 5548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:30.342697 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:30.342675 5548 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-7.ec2.internal" Oct 30 00:43:30.373795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373769 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.373957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373805 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.373957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373835 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.373957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373873 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.373957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373928 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.373957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.373935 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374008 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374090 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374135 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374139 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-env-overrides\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374166 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbsk6\" (UniqueName: \"kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374190 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.374218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374199 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374229 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374239 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7c43657-fe00-4520-9ee8-684223518dec-data-volume\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374271 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374300 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374312 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eeb58a8-a0fe-48f2-86a7-0e347a585522-tmp\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374317 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374338 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374369 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374421 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-tmp-dir\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374523 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4w7\" (UniqueName: \"kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.374535 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374533 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374612 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24vm\" (UniqueName: \"kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374613 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d248e8-94d3-40dd-baf3-9cfb15053504-metrics-certs\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374657 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043e9d07-b27e-4378-b055-c4d5bdcc29e3-iptables-alerter-script\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374710 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374783 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374825 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374877 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ed7102eb-69ac-4caf-8039-18466d995097-audit-log\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374888 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374921 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374965 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.374992 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375009 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375025 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375049 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.375081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375090 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ed2301-70f6-497f-b717-be08bdb686c4-tmp-dir\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375105 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375118 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375152 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375177 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375204 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njcm\" (UniqueName: \"kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375240 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375294 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eeb58a8-a0fe-48f2-86a7-0e347a585522-metrics-client-ca\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375331 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375347 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375363 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375396 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375413 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375437 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375455 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375472 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375487 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375503 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375641 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58ed2301-70f6-497f-b717-be08bdb686c4-metrics-tls\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.375686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375685 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375697 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-script-lib\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375731 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ed7102eb-69ac-4caf-8039-18466d995097-metrics-server-audit-profiles\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375750 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375755 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375781 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375812 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375830 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.375886 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqhg\" (UniqueName: \"kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376000 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e469533b-597a-4cf2-afd5-ad2fa146a978-serviceca\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376033 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376053 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed2301-70f6-497f-b717-be08bdb686c4-config-volume\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376087 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376103 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376133 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rqg\" (UniqueName: \"kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376182 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376223 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376252 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5mr\" (UniqueName: \"kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376277 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376305 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376329 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376353 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376395 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376488 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-client-ca-bundle\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376519 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.376537 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376548 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376573 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376632 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376632 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376681 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz2k\" (UniqueName: \"kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376720 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ab30191-af9f-4578-ae26-c57232aa5111-original-pull-secret\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376716 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376738 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376782 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376800 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdk5\" (UniqueName: \"kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376830 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376834 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376921 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376931 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376961 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.376993 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377018 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377115 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/368c928a-a45a-4446-942b-39bd53f20a04-agent-certs\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377131 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377167 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377194 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmnj\" (UniqueName: \"kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377230 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377246 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7c43657-fe00-4520-9ee8-684223518dec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377300 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lbv\" (UniqueName: \"kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377330 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377364 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377392 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377443 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-tls\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377444 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377501 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377537 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377563 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cni-binary-copy\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377572 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndl8\" (UniqueName: \"kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377597 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377615 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.377619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377638 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377667 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377721 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377755 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377772 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7c43657-fe00-4520-9ee8-684223518dec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377793 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377817 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ed7102eb-69ac-4caf-8039-18466d995097-secret-metrics-server-tls\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377825 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377884 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377928 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377947 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377956 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.377992 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378010 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378040 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378064 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-textfile\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378071 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fhd\" (UniqueName: \"kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378007 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378097 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378105 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-tuned\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378108 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378145 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378163 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378178 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378201 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwl4\" (UniqueName: \"kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378235 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378249 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378294 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378349 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378388 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378412 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378419 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-tmp\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378448 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zbd\" (UniqueName: \"kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378484 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378544 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378574 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378605 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378630 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378635 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378665 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378685 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378689 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40834bbb-90ae-486f-bbcb-ff665c3f313e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378701 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378719 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378748 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378827 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378887 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378903 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378910 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378921 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-daemon-config\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378930 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378954 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-grpc-tls\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378954 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.378976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.378993 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379027 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovn-node-metrics-cert\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379028 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379070 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379101 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379132 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1437846a-8b02-48e2-8963-d04d2432ecae-cert\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379150 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379140 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379172 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/368c928a-a45a-4446-942b-39bd53f20a04-konnectivity-ca\") pod \"konnectivity-agent-ts2gs\" (UID: \"368c928a-a45a-4446-942b-39bd53f20a04\") " pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379213 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379244 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379270 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a80b6429-b22b-4105-a70a-799faed22645-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379277 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379311 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379382 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llh49\" (UniqueName: \"kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379436 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379462 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7eeb58a8-a0fe-48f2-86a7-0e347a585522-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379497 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379531 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379566 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379592 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379622 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379653 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94j8\" (UniqueName: \"kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379780 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379812 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379837 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379899 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.379925 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.380005 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-ovnkube-config\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.380084 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-metrics-client-ca\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.380101 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49dd9e0f-977b-454c-8599-f6357642c0b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.380762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.380162 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a80b6429-b22b-4105-a70a-799faed22645-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.389118 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.389089 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4w7\" (UniqueName: \"kubernetes.io/projected/e469533b-597a-4cf2-afd5-ad2fa146a978-kube-api-access-cf4w7\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.389319 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.389302 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbsk6\" (UniqueName: \"kubernetes.io/projected/40834bbb-90ae-486f-bbcb-ff665c3f313e-kube-api-access-qbsk6\") pod \"kube-state-metrics-5dc8d7d89b-zmdjf\" (UID: \"40834bbb-90ae-486f-bbcb-ff665c3f313e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zmdjf" Oct 30 00:43:30.389780 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.389758 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24vm\" (UniqueName: \"kubernetes.io/projected/58ed2301-70f6-497f-b717-be08bdb686c4-kube-api-access-q24vm\") pod \"dns-default-8n2dz\" (UID: \"58ed2301-70f6-497f-b717-be08bdb686c4\") " pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:30.390108 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.390092 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njcm\" (UniqueName: \"kubernetes.io/projected/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-kube-api-access-2njcm\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.390624 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.390603 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqhg\" (UniqueName: \"kubernetes.io/projected/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-kube-api-access-tsqhg\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.391569 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.391542 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5mr\" (UniqueName: \"kubernetes.io/projected/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-kube-api-access-8r5mr\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.391775 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.391757 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz2k\" (UniqueName: \"kubernetes.io/projected/043e9d07-b27e-4378-b055-c4d5bdcc29e3-kube-api-access-khz2k\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.391966 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.391946 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdk5\" (UniqueName: \"kubernetes.io/projected/ed7102eb-69ac-4caf-8039-18466d995097-kube-api-access-jmdk5\") pod \"metrics-server-5f9c87f897-7zng6\" (UID: \"ed7102eb-69ac-4caf-8039-18466d995097\") " pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:30.391966 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.391955 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rqg\" (UniqueName: \"kubernetes.io/projected/1437846a-8b02-48e2-8963-d04d2432ecae-kube-api-access-j8rqg\") pod \"ingress-canary-gz2md\" (UID: \"1437846a-8b02-48e2-8963-d04d2432ecae\") " pod="openshift-ingress-canary/ingress-canary-gz2md" Oct 30 00:43:30.392146 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.392127 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmnj\" (UniqueName: \"kubernetes.io/projected/49dd9e0f-977b-454c-8599-f6357642c0b6-kube-api-access-skmnj\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.392258 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.392244 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lbv\" (UniqueName: \"kubernetes.io/projected/d7d248e8-94d3-40dd-baf3-9cfb15053504-kube-api-access-58lbv\") pod \"network-metrics-daemon-mrgqt\" (UID: \"d7d248e8-94d3-40dd-baf3-9cfb15053504\") " pod="openshift-multus/network-metrics-daemon-mrgqt" Oct 30 00:43:30.392326 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.392284 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndl8\" (UniqueName: \"kubernetes.io/projected/a7c43657-fe00-4520-9ee8-684223518dec-kube-api-access-jndl8\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.393151 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.393092 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zbd\" (UniqueName: \"kubernetes.io/projected/a80b6429-b22b-4105-a70a-799faed22645-kube-api-access-w6zbd\") pod \"openshift-state-metrics-7bd7489957-4cqjm\" (UID: \"a80b6429-b22b-4105-a70a-799faed22645\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4cqjm" Oct 30 00:43:30.393248 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.393165 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fhd\" (UniqueName: \"kubernetes.io/projected/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-kube-api-access-24fhd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.393477 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.393461 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwl4\" (UniqueName: \"kubernetes.io/projected/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kube-api-access-wjwl4\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.394081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.394059 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjfh\" (UniqueName: \"kubernetes.io/projected/ed92d686-6ba7-49de-b947-7178cb50b3c0-kube-api-access-vtjfh\") pod \"network-check-target-x9kzn\" (UID: \"ed92d686-6ba7-49de-b947-7178cb50b3c0\") " pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:30.394202 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.394136 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"alertmanager-main-0\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:30.394322 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.394306 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llh49\" (UniqueName: \"kubernetes.io/projected/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-kube-api-access-llh49\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.394402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.394389 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94j8\" (UniqueName: \"kubernetes.io/projected/7eeb58a8-a0fe-48f2-86a7-0e347a585522-kube-api-access-m94j8\") pod \"thanos-querier-5dbf9b4559-mh9jz\" (UID: \"7eeb58a8-a0fe-48f2-86a7-0e347a585522\") " pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:30.480420 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480347 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.480420 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480377 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.480420 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480397 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480419 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480443 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480469 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480472 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-device-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480493 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480497 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-conf-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480503 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-dbus\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480517 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480521 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-cnibin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480527 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480547 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-kubernetes\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480542 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480559 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-registration-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480568 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-slash\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480590 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480620 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480631 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480654 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-conf\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.480678 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480668 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-root\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480703 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480739 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480764 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480776 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480790 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480813 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480818 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-sys-fs\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480854 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ab30191-af9f-4578-ae26-c57232aa5111-kubelet-config\") pod \"global-pull-secret-syncer-ncxvc\" (UID: \"1ab30191-af9f-4578-ae26-c57232aa5111\") " pod="kube-system/global-pull-secret-syncer-ncxvc" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480879 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480908 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-systemd\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480914 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-var-lib-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480933 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-lib-modules\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480928 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480958 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480972 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.480988 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481015 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481029 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-kubelet\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481032 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e469533b-597a-4cf2-afd5-ad2fa146a978-host\") pod \"node-ca-kp9nh\" (UID: \"e469533b-597a-4cf2-afd5-ad2fa146a978\") " pod="openshift-image-registry/node-ca-kp9nh" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481036 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481039 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481066 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-ovn\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481083 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-socket-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481067 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6266e78f-7ca5-43bf-b3e6-6206fc45c4c9-hosts-file\") pod \"node-resolver-6qb5x\" (UID: \"6266e78f-7ca5-43bf-b3e6-6206fc45c4c9\") " pod="openshift-dns/node-resolver-6qb5x" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481082 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481122 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481126 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481141 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481160 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481179 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysconfig\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481185 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481196 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-bin\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481206 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481229 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481230 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-var-lib-kubelet\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481235 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-node-log\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481251 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481275 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-sys\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481282 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481276 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-cni-netd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481283 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-run\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481306 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481324 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-multus-socket-dir-parent\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481328 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481333 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-netns\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481354 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481370 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481385 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481403 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481423 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481430 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043e9d07-b27e-4378-b055-c4d5bdcc29e3-host-slash\") pod \"iptables-alerter-mst6g\" (UID: \"043e9d07-b27e-4378-b055-c4d5bdcc29e3\") " pod="openshift-network-operator/iptables-alerter-mst6g" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481432 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-etc-openvswitch\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481444 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481442 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-k8s-cni-cncf-io\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481451 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-bin\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481468 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-etc-kubernetes\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481477 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481487 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-host\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481469 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481516 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.481493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481539 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481555 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481574 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481577 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-modprobe-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481588 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d-node-exporter-wtmp\") pod \"node-exporter-8mh62\" (UID: \"aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d\") " pod="openshift-monitoring/node-exporter-8mh62" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481598 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481605 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-log-socket\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481624 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-var-lib-cni-multus\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481630 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481630 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-host-run-multus-certs\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481650 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481668 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481672 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-etc-sysctl-d\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481691 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481695 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-run-systemd\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481710 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481714 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-system-cni-dir\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481722 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f159ccf-7e4f-45d2-86e5-1b40e51df4b3-sys\") pod \"tuned-2gkzk\" (UID: \"8f159ccf-7e4f-45d2-86e5-1b40e51df4b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2gkzk" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481735 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481738 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-systemd-units\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481753 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481762 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54b428a8-331e-43e7-bfc1-fe7e0e6732fe-hostroot\") pod \"multus-g7tv9\" (UID: \"54b428a8-331e-43e7-bfc1-fe7e0e6732fe\") " pod="openshift-multus/multus-g7tv9" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481768 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481752 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7c43657-fe00-4520-9ee8-684223518dec-crio-socket\") pod \"insights-runtime-extractor-7kjw6\" (UID: \"a7c43657-fe00-4520-9ee8-684223518dec\") " pod="openshift-insights/insights-runtime-extractor-7kjw6" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481782 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-system-cni-dir\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481799 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-run-netns\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481806 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481823 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481836 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481888 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49dd9e0f-977b-454c-8599-f6357642c0b6-cnibin\") pod \"multus-additional-cni-plugins-k98jz\" (UID: \"49dd9e0f-977b-454c-8599-f6357642c0b6\") " pod="openshift-multus/multus-additional-cni-plugins-k98jz" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481902 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69fd86b2-89ac-4e07-8dd7-da1cf88785c3-host-kubelet\") pod \"ovnkube-node-tcd47\" (UID: \"69fd86b2-89ac-4e07-8dd7-da1cf88785c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:30.483188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:30.481917 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f2abc6d-480c-4b36-a525-ab1e04bf57c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89c5r\" (UID: \"7f2abc6d-480c-4b36-a525-ab1e04bf57c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89c5r" Oct 30 00:43:32.343392 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:32.343361 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8n2dz" Oct 30 00:43:32.677826 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:32.677799 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dbf9b4559-mh9jz" Oct 30 00:43:33.871829 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:33.871800 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:36.853902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:36.853873 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:36.854261 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:36.854077 5548 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:43:38.174001 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:38.173965 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:39.227644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:39.227614 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f9c87f897-7zng6" Oct 30 00:43:50.806401 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:50.806318 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tcd47" Oct 30 00:43:55.425996 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:43:55.425950 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:43:56.099985 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:56.099955 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ts2gs" Oct 30 00:43:58.176468 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.176441 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x9kzn" Oct 30 00:43:58.536748 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.536623 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:43:58.537205 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537170 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="alertmanager" containerID="cri-o://d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" gracePeriod=120 Oct 30 00:43:58.537324 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537204 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy" containerID="cri-o://eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" gracePeriod=120 Oct 30 00:43:58.537324 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537229 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="prom-label-proxy" containerID="cri-o://0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" gracePeriod=120 Oct 30 00:43:58.537324 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537265 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-web" containerID="cri-o://36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" gracePeriod=120 Oct 30 00:43:58.537475 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537273 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-metric" containerID="cri-o://4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" gracePeriod=120 Oct 30 00:43:58.537475 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:58.537259 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="config-reloader" containerID="cri-o://c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" gracePeriod=120 Oct 30 00:43:59.426681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426646 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" exitCode=0 Oct 30 00:43:59.426681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426676 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" exitCode=0 Oct 30 00:43:59.426681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426682 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" exitCode=0 Oct 30 00:43:59.426681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426688 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" exitCode=0 Oct 30 00:43:59.427262 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426722 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba"} Oct 30 00:43:59.427262 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426792 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083"} Oct 30 00:43:59.427262 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426819 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538"} Oct 30 00:43:59.427262 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.426845 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d"} Oct 30 00:43:59.818579 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.818549 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:43:59.893584 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893553 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.893756 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893618 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.893756 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893656 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.893756 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893688 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.893756 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893747 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893785 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893836 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893900 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893945 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.893988 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.894028 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.894057 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894226 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.894091 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") pod \"66681562-5027-4a5d-9e34-762f99b60322\" (UID: \"66681562-5027-4a5d-9e34-762f99b60322\") " Oct 30 00:43:59.894644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.894615 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:43:59.895803 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.895778 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:43:59.898727 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.898698 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:43:59.909238 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909208 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:43:59.909394 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909371 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.909492 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909220 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:43:59.909492 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909231 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.909618 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909524 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "kube-api-access-cvzfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:43:59.909618 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909576 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.909792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909771 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.909880 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.909838 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.912548 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.912528 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.920027 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.920002 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config" pod "66681562-5027-4a5d-9e34-762f99b60322" (UID: "66681562-5027-4a5d-9e34-762f99b60322"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995421 5548 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-config-out\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995448 5548 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-main-db\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995458 5548 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-web-config\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995467 5548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995477 5548 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-tls-assets\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995486 5548 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995494 5548 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995503 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66681562-5027-4a5d-9e34-762f99b60322-metrics-client-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995512 5548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995520 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzfw\" (UniqueName: \"kubernetes.io/projected/66681562-5027-4a5d-9e34-762f99b60322-kube-api-access-cvzfw\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995528 5548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995537 5548 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-cluster-tls-config\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:43:59.995795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:43:59.995545 5548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66681562-5027-4a5d-9e34-762f99b60322-secret-alertmanager-main-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:44:00.436302 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436268 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" exitCode=0 Oct 30 00:44:00.436302 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436293 5548 generic.go:356] "Generic (PLEG): container finished" podID="66681562-5027-4a5d-9e34-762f99b60322" containerID="36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" exitCode=0 Oct 30 00:44:00.436771 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436361 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.436771 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436355 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5"} Oct 30 00:44:00.436771 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436468 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563"} Oct 30 00:44:00.436771 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436514 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66681562-5027-4a5d-9e34-762f99b60322","Type":"ContainerDied","Data":"f888425ab2606f8e7e72b3ba28119cd2f776bc10b43f72b609f990542b544f39"} Oct 30 00:44:00.436771 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.436533 5548 scope.go:122] "RemoveContainer" containerID="0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" Oct 30 00:44:00.444450 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.444433 5548 scope.go:122] "RemoveContainer" containerID="4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" Oct 30 00:44:00.451250 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.451233 5548 scope.go:122] "RemoveContainer" containerID="eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" Oct 30 00:44:00.458059 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.458039 5548 scope.go:122] "RemoveContainer" containerID="36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" Oct 30 00:44:00.464717 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.464697 5548 scope.go:122] "RemoveContainer" containerID="c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" Oct 30 00:44:00.471379 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.471357 5548 scope.go:122] "RemoveContainer" containerID="d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" Oct 30 00:44:00.477894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.477879 5548 scope.go:122] "RemoveContainer" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" Oct 30 00:44:00.484483 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.484464 5548 scope.go:122] "RemoveContainer" containerID="0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" Oct 30 00:44:00.484725 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.484707 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba\": container with ID starting with 0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba not found: ID does not exist" containerID="0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" Oct 30 00:44:00.484772 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.484734 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba"} err="failed to get container status \"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba\": rpc error: code = NotFound desc = could not find container \"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba\": container with ID starting with 0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba not found: ID does not exist" Oct 30 00:44:00.484772 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.484764 5548 scope.go:122] "RemoveContainer" containerID="4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" Oct 30 00:44:00.484984 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.484969 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5\": container with ID starting with 4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5 not found: ID does not exist" containerID="4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" Oct 30 00:44:00.485032 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.484987 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5"} err="failed to get container status \"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5\": rpc error: code = NotFound desc = could not find container \"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5\": container with ID starting with 4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5 not found: ID does not exist" Oct 30 00:44:00.485032 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485000 5548 scope.go:122] "RemoveContainer" containerID="eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" Oct 30 00:44:00.485210 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.485190 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083\": container with ID starting with eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083 not found: ID does not exist" containerID="eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" Oct 30 00:44:00.485279 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485220 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083"} err="failed to get container status \"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083\": rpc error: code = NotFound desc = could not find container \"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083\": container with ID starting with eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083 not found: ID does not exist" Oct 30 00:44:00.485279 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485242 5548 scope.go:122] "RemoveContainer" containerID="36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" Oct 30 00:44:00.485503 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.485482 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563\": container with ID starting with 36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563 not found: ID does not exist" containerID="36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" Oct 30 00:44:00.485590 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485517 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563"} err="failed to get container status \"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563\": rpc error: code = NotFound desc = could not find container \"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563\": container with ID starting with 36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563 not found: ID does not exist" Oct 30 00:44:00.485590 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485530 5548 scope.go:122] "RemoveContainer" containerID="c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" Oct 30 00:44:00.485750 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.485733 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538\": container with ID starting with c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538 not found: ID does not exist" containerID="c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" Oct 30 00:44:00.485813 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485760 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538"} err="failed to get container status \"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538\": rpc error: code = NotFound desc = could not find container \"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538\": container with ID starting with c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538 not found: ID does not exist" Oct 30 00:44:00.485813 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.485782 5548 scope.go:122] "RemoveContainer" containerID="d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" Oct 30 00:44:00.486056 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.486038 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d\": container with ID starting with d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d not found: ID does not exist" containerID="d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" Oct 30 00:44:00.486129 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486063 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d"} err="failed to get container status \"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d\": rpc error: code = NotFound desc = could not find container \"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d\": container with ID starting with d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d not found: ID does not exist" Oct 30 00:44:00.486129 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486082 5548 scope.go:122] "RemoveContainer" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" Oct 30 00:44:00.486361 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:00.486345 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba\": container with ID starting with f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba not found: ID does not exist" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" Oct 30 00:44:00.486398 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486366 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba"} err="failed to get container status \"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba\": rpc error: code = NotFound desc = could not find container \"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba\": container with ID starting with f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba not found: ID does not exist" Oct 30 00:44:00.486398 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486378 5548 scope.go:122] "RemoveContainer" containerID="0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba" Oct 30 00:44:00.486602 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486588 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba"} err="failed to get container status \"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba\": rpc error: code = NotFound desc = could not find container \"0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba\": container with ID starting with 0db5f84e1dd69a8b4bf7ab4eb0f853788bf4480fe7dda1707891cd371128ebba not found: ID does not exist" Oct 30 00:44:00.486650 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486602 5548 scope.go:122] "RemoveContainer" containerID="4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5" Oct 30 00:44:00.486822 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486805 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5"} err="failed to get container status \"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5\": rpc error: code = NotFound desc = could not find container \"4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5\": container with ID starting with 4705ef00377ca4dd783fd63ec55a34528ac83f25ca62f38615592a1d5d5a95c5 not found: ID does not exist" Oct 30 00:44:00.486966 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.486824 5548 scope.go:122] "RemoveContainer" containerID="eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083" Oct 30 00:44:00.487107 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487087 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083"} err="failed to get container status \"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083\": rpc error: code = NotFound desc = could not find container \"eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083\": container with ID starting with eb702a342a9fe854506c66d154a020e40d38625209ae06b5bf30df9e6168d083 not found: ID does not exist" Oct 30 00:44:00.487158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487108 5548 scope.go:122] "RemoveContainer" containerID="36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563" Oct 30 00:44:00.487321 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487306 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563"} err="failed to get container status \"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563\": rpc error: code = NotFound desc = could not find container \"36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563\": container with ID starting with 36770633edbfb65e1356fb512158fc5640fbf8dd91ecbf4dcab6df4acd0f7563 not found: ID does not exist" Oct 30 00:44:00.487360 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487320 5548 scope.go:122] "RemoveContainer" containerID="c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538" Oct 30 00:44:00.487529 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487511 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538"} err="failed to get container status \"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538\": rpc error: code = NotFound desc = could not find container \"c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538\": container with ID starting with c7b12d6d50d44e6cd545286d4a222ec868aa2b59b64a800db7ec1237eb7f5538 not found: ID does not exist" Oct 30 00:44:00.487571 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487530 5548 scope.go:122] "RemoveContainer" containerID="d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d" Oct 30 00:44:00.487739 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487725 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d"} err="failed to get container status \"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d\": rpc error: code = NotFound desc = could not find container \"d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d\": container with ID starting with d5c586991fe388b73faf0631c8099b31f74946300e91910a7377b320293b347d not found: ID does not exist" Oct 30 00:44:00.487739 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487739 5548 scope.go:122] "RemoveContainer" containerID="f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba" Oct 30 00:44:00.487958 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.487943 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba"} err="failed to get container status \"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba\": rpc error: code = NotFound desc = could not find container \"f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba\": container with ID starting with f6ad729e75311ad8769bedf5b062dd0cffe6e34d6c3b4df1334ba9a506a232ba not found: ID does not exist" Oct 30 00:44:00.491532 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.491517 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:44:00.501474 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.501452 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:44:00.519350 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519331 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:44:00.519583 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519572 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="alertmanager" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519584 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="alertmanager" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519594 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519599 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519610 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="prom-label-proxy" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519615 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="prom-label-proxy" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519621 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="init-config-reloader" Oct 30 00:44:00.519625 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519625 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="init-config-reloader" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519633 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="config-reloader" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519637 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="config-reloader" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519646 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-web" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519650 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-web" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519656 5548 cpu_manager.go:414] "containerMap: removing container" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-metric" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519661 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-metric" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519705 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-web" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519713 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy-metric" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519721 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="alertmanager" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519727 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="kube-rbac-proxy" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519733 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="prom-label-proxy" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519737 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="init-config-reloader" Oct 30 00:44:00.519824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.519742 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="66681562-5027-4a5d-9e34-762f99b60322" containerName="config-reloader" Oct 30 00:44:00.553275 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.553246 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:44:00.553387 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.553353 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.566261 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566242 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:44:00.566360 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566276 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:44:00.566451 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566436 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:44:00.566516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566467 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:44:00.566516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566470 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:44:00.566516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566476 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:44:00.566912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566894 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:44:00.566972 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.566952 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:44:00.567114 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.567100 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n682p\"" Oct 30 00:44:00.576620 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.576606 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:44:00.600144 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600121 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600249 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600149 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600249 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600170 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600249 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600195 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rns\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-kube-api-access-x8rns\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600352 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600289 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600352 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600345 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600414 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600374 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600450 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600418 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600450 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600438 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-out\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600512 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600451 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600512 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600467 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600580 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600568 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.600617 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.600589 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-web-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.701694 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701637 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.701694 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701675 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-web-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.701694 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701695 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701818 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701847 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701913 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rns\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-kube-api-access-x8rns\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701940 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.701988 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702013 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702012 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702043 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702073 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-out\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702097 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702124 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702016 5548 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:44:00.702317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702298 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702617 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702586 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.702687 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.702657 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a9097c6-bbb4-4ccb-aeec-3922b227652c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.705308 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705278 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.705889 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705769 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.705889 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705819 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.705889 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705849 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-web-config\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.706037 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705982 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.706037 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.705968 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.706100 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.706031 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.706164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.706145 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a9097c6-bbb4-4ccb-aeec-3922b227652c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.706957 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.706937 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a9097c6-bbb4-4ccb-aeec-3922b227652c-config-out\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.718600 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.718581 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rns\" (UniqueName: \"kubernetes.io/projected/2a9097c6-bbb4-4ccb-aeec-3922b227652c-kube-api-access-x8rns\") pod \"alertmanager-main-0\" (UID: \"2a9097c6-bbb4-4ccb-aeec-3922b227652c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:00.962374 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:00.962294 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:44:01.261765 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:01.261690 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66681562-5027-4a5d-9e34-762f99b60322" path="/var/lib/kubelet/pods/66681562-5027-4a5d-9e34-762f99b60322/volumes" Oct 30 00:44:01.310373 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:01.310349 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:44:01.314495 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:44:01.314456 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9097c6_bbb4_4ccb_aeec_3922b227652c.slice/crio-f30a6cd7678855fdb3de084e8ac7c37d055d75f3cfa74ecae523039fbd5a6947 WatchSource:0}: Error finding container f30a6cd7678855fdb3de084e8ac7c37d055d75f3cfa74ecae523039fbd5a6947: Status 404 returned error can't find the container with id f30a6cd7678855fdb3de084e8ac7c37d055d75f3cfa74ecae523039fbd5a6947 Oct 30 00:44:01.445591 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:01.445561 5548 generic.go:356] "Generic (PLEG): container finished" podID="2a9097c6-bbb4-4ccb-aeec-3922b227652c" containerID="6f0de0ea8af135cba6ae46f77803ce22f69d6ca33245b7b47b14c687b5e743ba" exitCode=0 Oct 30 00:44:01.446041 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:01.445646 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerDied","Data":"6f0de0ea8af135cba6ae46f77803ce22f69d6ca33245b7b47b14c687b5e743ba"} Oct 30 00:44:01.446041 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:01.445758 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"f30a6cd7678855fdb3de084e8ac7c37d055d75f3cfa74ecae523039fbd5a6947"} Oct 30 00:44:02.453633 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453603 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"74c2af20b03b5fd397dfd76b416da3226b04d36d3177f0ffa8d39304732c402f"} Oct 30 00:44:02.454206 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453678 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"a2b0de24493bdcb9cf726f9df8af9f326c4f350567d5754e637570a2943f48e0"} Oct 30 00:44:02.454206 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453719 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"87fee7558a3a45f4dd97b44758046d0d19e8d610322fb8aebbe584a206d4e21e"} Oct 30 00:44:02.454206 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453765 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"e4af98ed435a915a9a4dbdcae0d364e2f02aabb470814fccf313a59c50c94f6e"} Oct 30 00:44:02.454206 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453808 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"eaf35e7145e04f5c0220e68c1c2f3b1ea4667c23254f739415b8b3d44a61251e"} Oct 30 00:44:02.454206 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.453849 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a9097c6-bbb4-4ccb-aeec-3922b227652c","Type":"ContainerStarted","Data":"c138268dc084b2d81ffdb387d242282484bdc1cccb90dcc4d3fb66ebb4eea1e7"} Oct 30 00:44:02.504641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.504592 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.491981 podStartE2EDuration="2.504578117s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.597117ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:02.504303823 +0000 UTC m=+33.346864415" watchObservedRunningTime="2026-10-30 00:44:02.504578117 +0000 UTC m=+33.347138741" Oct 30 00:44:02.514613 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.514588 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f4ccddd89-25bvh"] Oct 30 00:44:02.516833 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.516810 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.530641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.530517 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-srhpk\"" Oct 30 00:44:02.530641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.530527 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 00:44:02.530641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.530607 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 00:44:02.530641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.530609 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 00:44:02.530641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.530639 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 00:44:02.550469 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.550419 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f4ccddd89-25bvh"] Oct 30 00:44:02.616596 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.616563 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618154 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618246 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618305 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618350 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-metrics-client-ca\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618388 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-serving-certs-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618420 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-federate-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618476 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcsx\" (UniqueName: \"kubernetes.io/projected/11063556-5eff-4a8e-9005-5fe769c80fe7-kube-api-access-mkcsx\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.620902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.618507 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.627026 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.626995 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 00:44:02.719944 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.719874 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcsx\" (UniqueName: \"kubernetes.io/projected/11063556-5eff-4a8e-9005-5fe769c80fe7-kube-api-access-mkcsx\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.719944 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.719905 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.719944 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.719931 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.719965 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.719991 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.720009 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-metrics-client-ca\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720209 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.720026 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-serving-certs-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.720225 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-federate-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.720891 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.720835 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-metrics-client-ca\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.721232 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.721208 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-serving-certs-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.721427 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.721402 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.722717 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.722696 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.722893 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.722845 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-telemeter-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.723247 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.723230 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-federate-client-tls\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.723325 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.723304 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11063556-5eff-4a8e-9005-5fe769c80fe7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.736129 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.736107 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcsx\" (UniqueName: \"kubernetes.io/projected/11063556-5eff-4a8e-9005-5fe769c80fe7-kube-api-access-mkcsx\") pod \"telemeter-client-5f4ccddd89-25bvh\" (UID: \"11063556-5eff-4a8e-9005-5fe769c80fe7\") " pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:02.830263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:02.830230 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" Oct 30 00:44:03.147899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:03.147852 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f4ccddd89-25bvh"] Oct 30 00:44:03.152191 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:44:03.152167 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11063556_5eff_4a8e_9005_5fe769c80fe7.slice/crio-74ea2404812a63658f396bf1909b0c0dcf3d60f8514cafad8464f2fb7adc2cbd WatchSource:0}: Error finding container 74ea2404812a63658f396bf1909b0c0dcf3d60f8514cafad8464f2fb7adc2cbd: Status 404 returned error can't find the container with id 74ea2404812a63658f396bf1909b0c0dcf3d60f8514cafad8464f2fb7adc2cbd Oct 30 00:44:03.154181 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:03.154164 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:44:03.459382 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:03.459305 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" event={"ID":"11063556-5eff-4a8e-9005-5fe769c80fe7","Type":"ContainerStarted","Data":"74ea2404812a63658f396bf1909b0c0dcf3d60f8514cafad8464f2fb7adc2cbd"} Oct 30 00:44:05.469794 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:05.469754 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" event={"ID":"11063556-5eff-4a8e-9005-5fe769c80fe7","Type":"ContainerStarted","Data":"d1ec2da45bfd05eeaf47b78d66b461564f863ffb4543824afe1d8e0c50a17380"} Oct 30 00:44:05.470339 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:05.469815 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" event={"ID":"11063556-5eff-4a8e-9005-5fe769c80fe7","Type":"ContainerStarted","Data":"0bb04f273822014e17ad788b0d68f3b5eaf720dd65e3b551b02864740082c035"} Oct 30 00:44:05.470339 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:05.469835 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" event={"ID":"11063556-5eff-4a8e-9005-5fe769c80fe7","Type":"ContainerStarted","Data":"17f0b67f5f61612b5bcb5d35a7240a678d5748368984895da002d89e0fe06799"} Oct 30 00:44:05.525795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:44:05.525742 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f4ccddd89-25bvh" podStartSLOduration=2.000396795 podStartE2EDuration="3.525729744s" totalImagesPullingTime="1.525332949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:05.525262655 +0000 UTC m=+36.367823259" watchObservedRunningTime="2026-10-30 00:44:05.525729744 +0000 UTC m=+36.368290315" Oct 30 00:44:25.425246 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:25.425209 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:44:55.425970 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:44:55.425934 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:45:25.425764 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:45:25.425728 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:45:55.424799 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:45:55.424757 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:46:25.426420 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:46:25.426384 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:46:47.833790 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.833710 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk"] Oct 30 00:46:47.837603 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.837582 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:47.859347 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.859321 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:46:47.859347 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.859350 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-nbrxm\"" Oct 30 00:46:47.859534 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.859351 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:46:47.860289 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.860270 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk"] Oct 30 00:46:47.903250 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.903227 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5nz\" (UniqueName: \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:47.903400 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.903264 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:47.903400 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:47.903284 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.004169 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.004127 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.004169 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.004171 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.004398 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.004231 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5nz\" (UniqueName: \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.004605 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.004578 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.004681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.004590 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.020996 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.020970 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5nz\" (UniqueName: \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.153032 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.153002 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:46:48.442388 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:48.442315 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk"] Oct 30 00:46:48.446088 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:46:48.446053 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041cea8b_1895_4cad_8ad2_8cb8fd7d9a0d.slice/crio-82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a WatchSource:0}: Error finding container 82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a: Status 404 returned error can't find the container with id 82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a Oct 30 00:46:49.039643 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:49.039609 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" event={"ID":"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d","Type":"ContainerStarted","Data":"82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a"} Oct 30 00:46:54.066610 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:54.066571 5548 generic.go:356] "Generic (PLEG): container finished" podID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerID="02e00f5304f5059da829a8362f8e919e59e372646426199f15b873f32c18fb3e" exitCode=0 Oct 30 00:46:54.067134 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:54.066626 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" event={"ID":"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d","Type":"ContainerDied","Data":"02e00f5304f5059da829a8362f8e919e59e372646426199f15b873f32c18fb3e"} Oct 30 00:46:55.431497 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:46:55.431459 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:46:57.079818 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:57.079785 5548 generic.go:356] "Generic (PLEG): container finished" podID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerID="1d92d6e8678c5dd0060ce0b23b91ea32246ef05206216ec10d3ab6b158e65621" exitCode=0 Oct 30 00:46:57.080271 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:46:57.079887 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" event={"ID":"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d","Type":"ContainerDied","Data":"1d92d6e8678c5dd0060ce0b23b91ea32246ef05206216ec10d3ab6b158e65621"} Oct 30 00:47:03.105785 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:03.105752 5548 generic.go:356] "Generic (PLEG): container finished" podID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerID="a6811f22d55391b3e41cc62984dd39405c4d45fd143a5b7fc2a71d2a33e78975" exitCode=0 Oct 30 00:47:03.106214 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:03.105810 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" event={"ID":"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d","Type":"ContainerDied","Data":"a6811f22d55391b3e41cc62984dd39405c4d45fd143a5b7fc2a71d2a33e78975"} Oct 30 00:47:04.233192 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.233168 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:47:04.341938 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.341910 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\" (UniqueName: \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\") pod \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " Oct 30 00:47:04.342124 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.341956 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\") pod \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " Oct 30 00:47:04.342124 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.341978 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\") pod \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\" (UID: \"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d\") " Oct 30 00:47:04.342588 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.342561 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle" pod "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" (UID: "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:04.344238 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.344215 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz" pod "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" (UID: "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d"). InnerVolumeSpecName "kube-api-access-ht5nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:04.346519 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.346493 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util" pod "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" (UID: "041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:04.443493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.443464 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:04.443493 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.443489 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:04.443684 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:04.443499 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5nz\" (UniqueName: \"kubernetes.io/projected/041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d-kube-api-access-ht5nz\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:05.114222 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:05.114197 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" Oct 30 00:47:05.114222 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:05.114198 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6wfsk" event={"ID":"041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d","Type":"ContainerDied","Data":"82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a"} Oct 30 00:47:05.114468 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:05.114244 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cc615c07b557db39ed65a706e846c0b65f9cff597c4a5da28e914c007a741a" Oct 30 00:47:10.571220 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571184 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f"] Oct 30 00:47:10.571648 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571631 5548 cpu_manager.go:414] "containerMap: removing container" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="pull" Oct 30 00:47:10.571713 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571648 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="pull" Oct 30 00:47:10.571713 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571668 5548 cpu_manager.go:414] "containerMap: removing container" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="util" Oct 30 00:47:10.571713 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571677 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="util" Oct 30 00:47:10.571894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571718 5548 cpu_manager.go:414] "containerMap: removing container" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="extract" Oct 30 00:47:10.571894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571726 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="extract" Oct 30 00:47:10.571894 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.571800 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="041cea8b-1895-4cad-8ad2-8cb8fd7d9a0d" containerName="extract" Oct 30 00:47:10.573754 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.573734 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.586229 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.586205 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgkk\" (UniqueName: \"kubernetes.io/projected/7d107f10-ce01-4930-a4b4-8641802cd27e-kube-api-access-grgkk\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.586331 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.586292 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d107f10-ce01-4930-a4b4-8641802cd27e-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.590687 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.590670 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:47:10.590790 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.590775 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 00:47:10.596573 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.596555 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7sdvv\"" Oct 30 00:47:10.597460 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.597443 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f"] Oct 30 00:47:10.686840 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.686811 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d107f10-ce01-4930-a4b4-8641802cd27e-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.686994 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.686849 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgkk\" (UniqueName: \"kubernetes.io/projected/7d107f10-ce01-4930-a4b4-8641802cd27e-kube-api-access-grgkk\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.687168 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.687150 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d107f10-ce01-4930-a4b4-8641802cd27e-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.706252 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.706222 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgkk\" (UniqueName: \"kubernetes.io/projected/7d107f10-ce01-4930-a4b4-8641802cd27e-kube-api-access-grgkk\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-7mz9f\" (UID: \"7d107f10-ce01-4930-a4b4-8641802cd27e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:10.884035 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:10.883963 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" Oct 30 00:47:11.205043 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:11.205011 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f"] Oct 30 00:47:11.220695 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:11.220667 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d107f10_ce01_4930_a4b4_8641802cd27e.slice/crio-f1ec3ce370f646c471c3d39287b70b1cc413c0efcd86820c5dec119574c4a29e WatchSource:0}: Error finding container f1ec3ce370f646c471c3d39287b70b1cc413c0efcd86820c5dec119574c4a29e: Status 404 returned error can't find the container with id f1ec3ce370f646c471c3d39287b70b1cc413c0efcd86820c5dec119574c4a29e Oct 30 00:47:12.139286 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:12.139248 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" event={"ID":"7d107f10-ce01-4930-a4b4-8641802cd27e","Type":"ContainerStarted","Data":"f1ec3ce370f646c471c3d39287b70b1cc413c0efcd86820c5dec119574c4a29e"} Oct 30 00:47:14.150402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.150364 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" event={"ID":"7d107f10-ce01-4930-a4b4-8641802cd27e","Type":"ContainerStarted","Data":"74492114dbd4fea9da6b113f2f45704ad2f88d5904e865b953a5e8eba577ec64"} Oct 30 00:47:14.205616 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.205566 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-7mz9f" podStartSLOduration=2.121455814 podStartE2EDuration="4.205554077s" totalImagesPullingTime="2.084098263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:14.195241137 +0000 UTC m=+225.037801749" watchObservedRunningTime="2026-10-30 00:47:14.205554077 +0000 UTC m=+225.048114669" Oct 30 00:47:14.716046 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.716018 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz"] Oct 30 00:47:14.718642 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.718620 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.738130 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.738100 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:47:14.749402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.747592 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rnl7z\"" Oct 30 00:47:14.749402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.747625 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:47:14.749402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.748765 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz"] Oct 30 00:47:14.816027 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.815992 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.816191 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.816060 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqjm\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-kube-api-access-pqqjm\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.917223 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.917189 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.917385 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.917243 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqjm\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-kube-api-access-pqqjm\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.935193 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.935162 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:14.935363 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:14.935345 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqjm\" (UniqueName: \"kubernetes.io/projected/a2224fe1-ff16-4487-aba1-1f24e8f3d7d1-kube-api-access-pqqjm\") pod \"cert-manager-webhook-5dbbbf75cc-qchzz\" (UID: \"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:15.041343 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.041268 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:15.336164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.336075 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz"] Oct 30 00:47:15.340294 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:15.340265 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2224fe1_ff16_4487_aba1_1f24e8f3d7d1.slice/crio-f833baafad97b2c163214b128e28e99f5a6238453e661157e623e44ecc0d955c WatchSource:0}: Error finding container f833baafad97b2c163214b128e28e99f5a6238453e661157e623e44ecc0d955c: Status 404 returned error can't find the container with id f833baafad97b2c163214b128e28e99f5a6238453e661157e623e44ecc0d955c Oct 30 00:47:15.419437 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.419403 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9"] Oct 30 00:47:15.422462 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.422439 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.423648 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.423627 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.423757 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.423663 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.423757 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.423682 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rnk\" (UniqueName: \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.434534 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.434513 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:47:15.434534 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.434521 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:47:15.434680 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.434596 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-967fv\"" Oct 30 00:47:15.445263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.445238 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9"] Oct 30 00:47:15.524128 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.524088 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.524314 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.524140 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.524314 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.524163 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rnk\" (UniqueName: \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.524492 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.524472 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.524555 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.524530 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.544340 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.544313 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rnk\" (UniqueName: \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:15.735452 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:15.735417 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:16.025975 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:16.025893 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9"] Oct 30 00:47:16.029336 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:16.029294 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218f1a26_5a7c_498a_9343_f4d634a0127c.slice/crio-4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927 WatchSource:0}: Error finding container 4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927: Status 404 returned error can't find the container with id 4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927 Oct 30 00:47:16.167441 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:16.167407 5548 generic.go:356] "Generic (PLEG): container finished" podID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerID="b7254b0501f6d5f0eaefed11b736594a828e617468d5a2c28cb158a5ada02d6f" exitCode=0 Oct 30 00:47:16.167587 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:16.167445 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" event={"ID":"218f1a26-5a7c-498a-9343-f4d634a0127c","Type":"ContainerDied","Data":"b7254b0501f6d5f0eaefed11b736594a828e617468d5a2c28cb158a5ada02d6f"} Oct 30 00:47:16.167587 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:16.167518 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" event={"ID":"218f1a26-5a7c-498a-9343-f4d634a0127c","Type":"ContainerStarted","Data":"4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927"} Oct 30 00:47:16.169217 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:16.169191 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" event={"ID":"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1","Type":"ContainerStarted","Data":"f833baafad97b2c163214b128e28e99f5a6238453e661157e623e44ecc0d955c"} Oct 30 00:47:18.180947 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:18.180897 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" event={"ID":"a2224fe1-ff16-4487-aba1-1f24e8f3d7d1","Type":"ContainerStarted","Data":"bed59af176e18cbf09c086fb033726ccf5545de23b7e0953da38c00e9769544b"} Oct 30 00:47:18.181391 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:18.180984 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:18.213874 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:18.213800 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" podStartSLOduration=1.581750833 podStartE2EDuration="4.213787581s" totalImagesPullingTime="2.632036748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:18.21328388 +0000 UTC m=+229.055844492" watchObservedRunningTime="2026-10-30 00:47:18.213787581 +0000 UTC m=+229.056348173" Oct 30 00:47:19.186814 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:19.186780 5548 generic.go:356] "Generic (PLEG): container finished" podID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerID="7b014c45634abdd0544c61df869ab98bb52d5ce8c7139769caa6eec8dc26a557" exitCode=0 Oct 30 00:47:19.187271 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:19.186854 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" event={"ID":"218f1a26-5a7c-498a-9343-f4d634a0127c","Type":"ContainerDied","Data":"7b014c45634abdd0544c61df869ab98bb52d5ce8c7139769caa6eec8dc26a557"} Oct 30 00:47:20.193543 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:20.193512 5548 generic.go:356] "Generic (PLEG): container finished" podID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerID="d1d07b8966beb66a70f7369598d1119e75fefbe8e92fa1189d97611863d5fcee" exitCode=0 Oct 30 00:47:20.194000 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:20.193573 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" event={"ID":"218f1a26-5a7c-498a-9343-f4d634a0127c","Type":"ContainerDied","Data":"d1d07b8966beb66a70f7369598d1119e75fefbe8e92fa1189d97611863d5fcee"} Oct 30 00:47:21.342875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.342837 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:21.368896 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.368848 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\") pod \"218f1a26-5a7c-498a-9343-f4d634a0127c\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " Oct 30 00:47:21.369042 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.368965 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\") pod \"218f1a26-5a7c-498a-9343-f4d634a0127c\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " Oct 30 00:47:21.369042 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.369004 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\" (UniqueName: \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\") pod \"218f1a26-5a7c-498a-9343-f4d634a0127c\" (UID: \"218f1a26-5a7c-498a-9343-f4d634a0127c\") " Oct 30 00:47:21.369441 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.369391 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle" pod "218f1a26-5a7c-498a-9343-f4d634a0127c" (UID: "218f1a26-5a7c-498a-9343-f4d634a0127c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:21.371469 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.371443 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk" pod "218f1a26-5a7c-498a-9343-f4d634a0127c" (UID: "218f1a26-5a7c-498a-9343-f4d634a0127c"). InnerVolumeSpecName "kube-api-access-s7rnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:21.373527 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.373505 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util" pod "218f1a26-5a7c-498a-9343-f4d634a0127c" (UID: "218f1a26-5a7c-498a-9343-f4d634a0127c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:21.470228 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.470132 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:21.470228 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.470176 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rnk\" (UniqueName: \"kubernetes.io/projected/218f1a26-5a7c-498a-9343-f4d634a0127c-kube-api-access-s7rnk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:21.470228 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:21.470188 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/218f1a26-5a7c-498a-9343-f4d634a0127c-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:22.202643 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:22.202606 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" event={"ID":"218f1a26-5a7c-498a-9343-f4d634a0127c","Type":"ContainerDied","Data":"4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927"} Oct 30 00:47:22.202825 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:22.202660 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3718869a59d8b011a281fd84069daf23312a948f50200962b9e39a1bfca927" Oct 30 00:47:22.202825 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:22.202624 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85xr4n9" Oct 30 00:47:24.189293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:24.189264 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qchzz" Oct 30 00:47:25.425669 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:47:25.425631 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:47:26.232992 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.232960 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jpjp2"] Oct 30 00:47:26.233346 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233330 5548 cpu_manager.go:414] "containerMap: removing container" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="util" Oct 30 00:47:26.233421 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233347 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="util" Oct 30 00:47:26.233421 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233361 5548 cpu_manager.go:414] "containerMap: removing container" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="extract" Oct 30 00:47:26.233421 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233366 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="extract" Oct 30 00:47:26.233421 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233383 5548 cpu_manager.go:414] "containerMap: removing container" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="pull" Oct 30 00:47:26.233421 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233388 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="pull" Oct 30 00:47:26.233601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.233438 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="218f1a26-5a7c-498a-9343-f4d634a0127c" containerName="extract" Oct 30 00:47:26.236932 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.236913 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.258590 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.258566 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-whrlt\"" Oct 30 00:47:26.259622 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.259598 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jpjp2"] Oct 30 00:47:26.309781 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.309751 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jc2\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-kube-api-access-29jc2\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.309946 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.309798 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-bound-sa-token\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.411121 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.411090 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-bound-sa-token\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.411256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.411158 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jc2\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-kube-api-access-29jc2\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.427481 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.427447 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-bound-sa-token\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.427813 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.427558 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jc2\" (UniqueName: \"kubernetes.io/projected/c8a15f5a-810f-4dad-8c8e-36a37b76b4f4-kube-api-access-29jc2\") pod \"cert-manager-597b95cb84-jpjp2\" (UID: \"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4\") " pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.547970 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.547895 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jpjp2" Oct 30 00:47:26.837547 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:26.837480 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jpjp2"] Oct 30 00:47:26.841119 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:26.841078 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a15f5a_810f_4dad_8c8e_36a37b76b4f4.slice/crio-72b1f0dade0937447d3b4b9972c66987fb39114db4022b3cabe26a3332e50dcb WatchSource:0}: Error finding container 72b1f0dade0937447d3b4b9972c66987fb39114db4022b3cabe26a3332e50dcb: Status 404 returned error can't find the container with id 72b1f0dade0937447d3b4b9972c66987fb39114db4022b3cabe26a3332e50dcb Oct 30 00:47:27.226483 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:27.226450 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jpjp2" event={"ID":"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4","Type":"ContainerStarted","Data":"e5a3cf3b8d14c3056d87afb00cdcf14fee5c65be3bed98de9f00cc1e6072458f"} Oct 30 00:47:27.226783 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:27.226491 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jpjp2" event={"ID":"c8a15f5a-810f-4dad-8c8e-36a37b76b4f4","Type":"ContainerStarted","Data":"72b1f0dade0937447d3b4b9972c66987fb39114db4022b3cabe26a3332e50dcb"} Oct 30 00:47:27.259510 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:27.259447 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jpjp2" podStartSLOduration=1.259428812 podStartE2EDuration="1.259428812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:27.258641845 +0000 UTC m=+238.101202438" watchObservedRunningTime="2026-10-30 00:47:27.259428812 +0000 UTC m=+238.101989405" Oct 30 00:47:28.223505 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.223468 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-7lftj"] Oct 30 00:47:28.229914 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.229892 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.256843 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.256815 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 00:47:28.256843 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.256819 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:47:28.257081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.256849 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ndhqm\"" Oct 30 00:47:28.257753 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.257734 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-7lftj"] Oct 30 00:47:28.328952 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.328905 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-tmp\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.329161 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.329131 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b8v6\" (UniqueName: \"kubernetes.io/projected/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-kube-api-access-5b8v6\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.430354 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.430317 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b8v6\" (UniqueName: \"kubernetes.io/projected/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-kube-api-access-5b8v6\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.430354 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.430361 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-tmp\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.430794 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.430774 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-tmp\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.447710 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.447676 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b8v6\" (UniqueName: \"kubernetes.io/projected/9efc0799-e6b0-478e-996a-e7bf6e1f9ac2-kube-api-access-5b8v6\") pod \"openshift-lws-operator-8465994787-7lftj\" (UID: \"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.542635 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.542533 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" Oct 30 00:47:28.834440 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:28.834374 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-7lftj"] Oct 30 00:47:28.838150 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:28.838121 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9efc0799_e6b0_478e_996a_e7bf6e1f9ac2.slice/crio-8cdc791982fbaaf4552b0bd50d0a5a8d75dc657ddd43fdeaa12d70ea1b0c2690 WatchSource:0}: Error finding container 8cdc791982fbaaf4552b0bd50d0a5a8d75dc657ddd43fdeaa12d70ea1b0c2690: Status 404 returned error can't find the container with id 8cdc791982fbaaf4552b0bd50d0a5a8d75dc657ddd43fdeaa12d70ea1b0c2690 Oct 30 00:47:29.236217 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:29.236186 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" event={"ID":"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2","Type":"ContainerStarted","Data":"8cdc791982fbaaf4552b0bd50d0a5a8d75dc657ddd43fdeaa12d70ea1b0c2690"} Oct 30 00:47:31.245786 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:31.245751 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" event={"ID":"9efc0799-e6b0-478e-996a-e7bf6e1f9ac2","Type":"ContainerStarted","Data":"2f64984742a5f8502d94cd8210cc02887d835096299983203f7f36f0818776ce"} Oct 30 00:47:31.288981 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:31.288938 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-7lftj" podStartSLOduration=1.3868309779999999 podStartE2EDuration="3.288925937s" totalImagesPullingTime="1.902094959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:31.288250398 +0000 UTC m=+242.130810990" watchObservedRunningTime="2026-10-30 00:47:31.288925937 +0000 UTC m=+242.131486530" Oct 30 00:47:34.467517 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.467487 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf"] Oct 30 00:47:34.488710 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.488679 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf"] Oct 30 00:47:34.488888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.488795 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.510351 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.510328 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:47:34.510453 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.510371 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:47:34.510453 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.510371 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-tmlm7\"" Oct 30 00:47:34.584891 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.584849 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.585039 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.584901 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbkd\" (UniqueName: \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.585039 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.585024 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.686157 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.686132 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.686305 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.686194 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.686305 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.686230 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbkd\" (UniqueName: \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.686628 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.686609 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.686670 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.686641 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.702465 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.702434 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbkd\" (UniqueName: \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:34.843204 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:34.843134 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:35.147883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:35.147829 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf"] Oct 30 00:47:35.151548 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:35.151517 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5505e2d3_c252_4a12_8cdd_18c54e6d847b.slice/crio-0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d WatchSource:0}: Error finding container 0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d: Status 404 returned error can't find the container with id 0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d Oct 30 00:47:35.263943 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:35.263905 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerStarted","Data":"a43caa651cd90339322468fb12bd6344f084bb9808eff770186203a1aa175abf"} Oct 30 00:47:35.264106 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:35.263975 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerStarted","Data":"0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d"} Oct 30 00:47:36.269605 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:36.269568 5548 generic.go:356] "Generic (PLEG): container finished" podID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerID="a43caa651cd90339322468fb12bd6344f084bb9808eff770186203a1aa175abf" exitCode=0 Oct 30 00:47:36.270091 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:36.269657 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerDied","Data":"a43caa651cd90339322468fb12bd6344f084bb9808eff770186203a1aa175abf"} Oct 30 00:47:37.275971 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:37.275942 5548 generic.go:356] "Generic (PLEG): container finished" podID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerID="3f4f130a28c119d33cb37c930e85e3169ac9a2f82a4cc6df872c00d410fa97b2" exitCode=0 Oct 30 00:47:37.276332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:37.276037 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerDied","Data":"3f4f130a28c119d33cb37c930e85e3169ac9a2f82a4cc6df872c00d410fa97b2"} Oct 30 00:47:38.282188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:38.282153 5548 generic.go:356] "Generic (PLEG): container finished" podID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerID="99bd1a1cad0e6e8d6052d9b8efd9137481cdf19959e942d5d3fcb4f18e1b2b5d" exitCode=0 Oct 30 00:47:38.282571 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:38.282229 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerDied","Data":"99bd1a1cad0e6e8d6052d9b8efd9137481cdf19959e942d5d3fcb4f18e1b2b5d"} Oct 30 00:47:39.407644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.407621 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:39.526211 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.526174 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\" (UniqueName: \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\") pod \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " Oct 30 00:47:39.526211 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.526215 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\") pod \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " Oct 30 00:47:39.526433 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.526252 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\") pod \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\" (UID: \"5505e2d3-c252-4a12-8cdd-18c54e6d847b\") " Oct 30 00:47:39.526902 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.526841 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle" pod "5505e2d3-c252-4a12-8cdd-18c54e6d847b" (UID: "5505e2d3-c252-4a12-8cdd-18c54e6d847b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:39.528473 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.528448 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd" pod "5505e2d3-c252-4a12-8cdd-18c54e6d847b" (UID: "5505e2d3-c252-4a12-8cdd-18c54e6d847b"). InnerVolumeSpecName "kube-api-access-kqbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:39.531591 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.531555 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util" pod "5505e2d3-c252-4a12-8cdd-18c54e6d847b" (UID: "5505e2d3-c252-4a12-8cdd-18c54e6d847b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:47:39.627099 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.627020 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbkd\" (UniqueName: \"kubernetes.io/projected/5505e2d3-c252-4a12-8cdd-18c54e6d847b-kube-api-access-kqbkd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:39.627099 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.627047 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:39.627099 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:39.627058 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5505e2d3-c252-4a12-8cdd-18c54e6d847b-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:47:40.291853 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:40.291825 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" Oct 30 00:47:40.292029 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:40.291822 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n22sf" event={"ID":"5505e2d3-c252-4a12-8cdd-18c54e6d847b","Type":"ContainerDied","Data":"0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d"} Oct 30 00:47:40.292029 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:40.291976 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c36ff6560bb63738ae9d9ec55379964f0073184c25d1ce4e567322008bd940d" Oct 30 00:47:49.011565 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.011527 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588"] Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.011961 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="pull" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.011980 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="pull" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.011996 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="extract" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.012001 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="extract" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.012016 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="util" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.012023 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="util" Oct 30 00:47:49.012135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.012104 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5505e2d3-c252-4a12-8cdd-18c54e6d847b" containerName="extract" Oct 30 00:47:49.042617 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.042579 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588"] Oct 30 00:47:49.042811 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.042743 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.119551 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.119525 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 00:47:49.119704 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.119643 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 00:47:49.119704 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.119689 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 00:47:49.119776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.119719 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-srkcr\"" Oct 30 00:47:49.207293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.207265 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr24s\" (UniqueName: \"kubernetes.io/projected/3c7744ce-9827-453a-a165-492cc54f4c7c-kube-api-access-pr24s\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.207440 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.207296 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c7744ce-9827-453a-a165-492cc54f4c7c-manager-config\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.207440 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.207327 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.207440 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.207419 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.308323 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.308255 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.308323 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.308318 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr24s\" (UniqueName: \"kubernetes.io/projected/3c7744ce-9827-453a-a165-492cc54f4c7c-kube-api-access-pr24s\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.308510 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.308337 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c7744ce-9827-453a-a165-492cc54f4c7c-manager-config\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.308510 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.308368 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.309145 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.309047 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c7744ce-9827-453a-a165-492cc54f4c7c-manager-config\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.311036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.311014 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.311115 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.311017 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c7744ce-9827-453a-a165-492cc54f4c7c-cert\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.324850 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.324818 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr24s\" (UniqueName: \"kubernetes.io/projected/3c7744ce-9827-453a-a165-492cc54f4c7c-kube-api-access-pr24s\") pod \"lws-controller-manager-85798cdd6b-5k588\" (UID: \"3c7744ce-9827-453a-a165-492cc54f4c7c\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.376061 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.376028 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:49.684417 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.684204 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588"] Oct 30 00:47:49.686336 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:49.686310 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7744ce_9827_453a_a165_492cc54f4c7c.slice/crio-0b806cb878d93cc7f9d9958c7426ff02bd3b9bf4c6bfba53d03aeda6fa9a70db WatchSource:0}: Error finding container 0b806cb878d93cc7f9d9958c7426ff02bd3b9bf4c6bfba53d03aeda6fa9a70db: Status 404 returned error can't find the container with id 0b806cb878d93cc7f9d9958c7426ff02bd3b9bf4c6bfba53d03aeda6fa9a70db Oct 30 00:47:49.781496 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.781458 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj"] Oct 30 00:47:49.785086 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.785063 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.799735 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.799712 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:47:49.799885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.799846 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:47:49.800036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.799943 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:47:49.800036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.799950 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:47:49.800036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.799997 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:47:49.800576 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.800559 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h7rmk\"" Oct 30 00:47:49.846101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.846067 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj"] Oct 30 00:47:49.912461 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912426 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.912632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912470 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.912632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912564 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.912718 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912635 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.912718 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912654 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:49.912718 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:49.912672 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btpw\" (UniqueName: \"kubernetes.io/projected/cda57833-6b29-49b5-a0f8-7bf80508dd8a-kube-api-access-5btpw\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.013601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013519 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.013601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013565 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.013601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013589 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.014153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013617 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.014153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013635 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.014153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.013649 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btpw\" (UniqueName: \"kubernetes.io/projected/cda57833-6b29-49b5-a0f8-7bf80508dd8a-kube-api-access-5btpw\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.014153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.014116 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.016406 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.016380 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.016526 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.016480 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda57833-6b29-49b5-a0f8-7bf80508dd8a-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.016526 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.016491 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.016688 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.016669 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cda57833-6b29-49b5-a0f8-7bf80508dd8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.031274 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.031249 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btpw\" (UniqueName: \"kubernetes.io/projected/cda57833-6b29-49b5-a0f8-7bf80508dd8a-kube-api-access-5btpw\") pod \"opendatahub-operator-controller-manager-84dd548884-487tj\" (UID: \"cda57833-6b29-49b5-a0f8-7bf80508dd8a\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.099298 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.099271 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:50.333920 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.333836 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" event={"ID":"3c7744ce-9827-453a-a165-492cc54f4c7c","Type":"ContainerStarted","Data":"0b806cb878d93cc7f9d9958c7426ff02bd3b9bf4c6bfba53d03aeda6fa9a70db"} Oct 30 00:47:50.444319 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:50.444279 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj"] Oct 30 00:47:50.446585 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:47:50.446558 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda57833_6b29_49b5_a0f8_7bf80508dd8a.slice/crio-cde631749a2228a45f5e716950a3a8c0c1c76add8425aad249813f2e266d086a WatchSource:0}: Error finding container cde631749a2228a45f5e716950a3a8c0c1c76add8425aad249813f2e266d086a: Status 404 returned error can't find the container with id cde631749a2228a45f5e716950a3a8c0c1c76add8425aad249813f2e266d086a Oct 30 00:47:51.346978 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:51.346894 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" event={"ID":"cda57833-6b29-49b5-a0f8-7bf80508dd8a","Type":"ContainerStarted","Data":"cde631749a2228a45f5e716950a3a8c0c1c76add8425aad249813f2e266d086a"} Oct 30 00:47:52.354441 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:52.354401 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" event={"ID":"3c7744ce-9827-453a-a165-492cc54f4c7c","Type":"ContainerStarted","Data":"69da39974e768dd1fdb042261dcd572a85c0059ca1bb852a0763bad430db91c5"} Oct 30 00:47:52.354903 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:52.354554 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:47:52.393708 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:52.393647 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" podStartSLOduration=2.566594635 podStartE2EDuration="4.393629306s" totalImagesPullingTime="1.827034671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:52.392034685 +0000 UTC m=+263.234595278" watchObservedRunningTime="2026-10-30 00:47:52.393629306 +0000 UTC m=+263.236189900" Oct 30 00:47:54.363320 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:54.363284 5548 generic.go:356] "Generic (PLEG): container finished" podID="cda57833-6b29-49b5-a0f8-7bf80508dd8a" containerID="140d7eb76b89cc3331c72289dbc8b59cda0a33d492b5f64fc7681a8e6ec1e74b" exitCode=0 Oct 30 00:47:54.363738 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:54.363367 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" event={"ID":"cda57833-6b29-49b5-a0f8-7bf80508dd8a","Type":"ContainerDied","Data":"140d7eb76b89cc3331c72289dbc8b59cda0a33d492b5f64fc7681a8e6ec1e74b"} Oct 30 00:47:55.369284 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:55.369249 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" event={"ID":"cda57833-6b29-49b5-a0f8-7bf80508dd8a","Type":"ContainerStarted","Data":"ba9b8ae2c607361f60851b201c279f3def2a4552ef84034fcf068412626ae7ef"} Oct 30 00:47:55.369675 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:55.369366 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:47:55.426504 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:47:55.426463 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:47:55.434447 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:47:55.434399 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" podStartSLOduration=2.760556776 podStartE2EDuration="6.434385263s" totalImagesPullingTime="3.187002856s" totalInitContainerRuntime="486.825631ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:55.424269948 +0000 UTC m=+266.266830559" watchObservedRunningTime="2026-10-30 00:47:55.434385263 +0000 UTC m=+266.276945856" Oct 30 00:48:03.360648 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:03.360614 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-5k588" Oct 30 00:48:06.376368 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:06.376335 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-487tj" Oct 30 00:48:14.328263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.328223 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-d777n"] Oct 30 00:48:14.332034 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.332011 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:14.354617 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.354567 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 00:48:14.354803 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.354691 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-s75z6\"" Oct 30 00:48:14.356406 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.356383 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-d777n"] Oct 30 00:48:14.414998 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.414961 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:14.415141 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.415086 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfdx\" (UniqueName: \"kubernetes.io/projected/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-kube-api-access-vlfdx\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:14.515693 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.515656 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfdx\" (UniqueName: \"kubernetes.io/projected/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-kube-api-access-vlfdx\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:14.515853 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.515746 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:14.515952 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:48:14.515931 5548 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 00:48:14.516012 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:48:14.516002 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs podName:a02e4f48-ac1a-4cfe-b406-1cd83d454a7c nodeName:}" failed. No retries permitted until 2026-10-30 00:48:15.015988246 +0000 UTC m=+285.858548816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs") pod "odh-observability-584f99c788-d777n" (UID: "a02e4f48-ac1a-4cfe-b406-1cd83d454a7c") : secret "odh-observability-webhook-cert" not found Oct 30 00:48:14.533793 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:14.533771 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfdx\" (UniqueName: \"kubernetes.io/projected/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-kube-api-access-vlfdx\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:15.020447 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:15.020414 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:15.023063 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:15.023034 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/a02e4f48-ac1a-4cfe-b406-1cd83d454a7c-webhook-certs\") pod \"odh-observability-584f99c788-d777n\" (UID: \"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c\") " pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:15.244549 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:15.244514 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:15.547077 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:15.547046 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-d777n"] Oct 30 00:48:15.550566 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:48:15.550539 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02e4f48_ac1a_4cfe_b406_1cd83d454a7c.slice/crio-52d58ee61221e00c4e62fd8a2b52bfe37d9715d2f7a6376c3373990480b3c7f0 WatchSource:0}: Error finding container 52d58ee61221e00c4e62fd8a2b52bfe37d9715d2f7a6376c3373990480b3c7f0: Status 404 returned error can't find the container with id 52d58ee61221e00c4e62fd8a2b52bfe37d9715d2f7a6376c3373990480b3c7f0 Oct 30 00:48:16.202487 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.202449 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g"] Oct 30 00:48:16.206408 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.206380 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.222297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.222229 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4qrzd\"" Oct 30 00:48:16.222297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.222246 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:48:16.231698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231660 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5mj\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-kube-api-access-wg5mj\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.231838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231713 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.231838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231791 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.231838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231826 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.232062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231897 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.232062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231941 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.232062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.231986 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.232062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.232017 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.232062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.232054 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.234652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.234426 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g"] Oct 30 00:48:16.234652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.234549 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:48:16.234652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.234624 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:48:16.333403 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333352 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5mj\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-kube-api-access-wg5mj\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333612 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333412 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333612 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333447 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333612 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333481 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333612 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333521 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333612 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333566 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333924 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333612 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333924 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333657 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.333924 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.333693 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.334216 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.334185 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.334517 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.334489 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.334761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.334637 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.334761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.334640 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.334761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.334686 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.336570 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.336538 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.337349 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.337268 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.350875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.350811 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.351037 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.351011 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5mj\" (UniqueName: \"kubernetes.io/projected/54ef06e3-eb02-4a98-ac7a-910a12c8462b-kube-api-access-wg5mj\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g\" (UID: \"54ef06e3-eb02-4a98-ac7a-910a12c8462b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.461591 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.461498 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-d777n" event={"ID":"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c","Type":"ContainerStarted","Data":"52d58ee61221e00c4e62fd8a2b52bfe37d9715d2f7a6376c3373990480b3c7f0"} Oct 30 00:48:16.526430 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.526399 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:16.845345 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:16.845316 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g"] Oct 30 00:48:16.845836 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:48:16.845813 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ef06e3_eb02_4a98_ac7a_910a12c8462b.slice/crio-9d35c12721b518abdc7bb12c9c522c1d42d3e42ade844bf3586f4ca40aba6bbd WatchSource:0}: Error finding container 9d35c12721b518abdc7bb12c9c522c1d42d3e42ade844bf3586f4ca40aba6bbd: Status 404 returned error can't find the container with id 9d35c12721b518abdc7bb12c9c522c1d42d3e42ade844bf3586f4ca40aba6bbd Oct 30 00:48:17.467873 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:17.467834 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" event={"ID":"54ef06e3-eb02-4a98-ac7a-910a12c8462b","Type":"ContainerStarted","Data":"9d35c12721b518abdc7bb12c9c522c1d42d3e42ade844bf3586f4ca40aba6bbd"} Oct 30 00:48:18.477484 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:18.477448 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-d777n" event={"ID":"a02e4f48-ac1a-4cfe-b406-1cd83d454a7c","Type":"ContainerStarted","Data":"a33b5cc11e549dad574e2a8fad695e010246597eec63a4f0fdc838c0b97d7e80"} Oct 30 00:48:18.477988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:18.477564 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:18.510269 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:18.510213 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-d777n" podStartSLOduration=2.148302564 podStartE2EDuration="4.510199338s" totalImagesPullingTime="2.361896774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:18.510186263 +0000 UTC m=+289.352746856" watchObservedRunningTime="2026-10-30 00:48:18.510199338 +0000 UTC m=+289.352759931" Oct 30 00:48:19.609116 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:19.609084 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:48:19.609382 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:19.609155 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:48:19.609382 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:19.609184 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:48:20.487760 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:20.487725 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" event={"ID":"54ef06e3-eb02-4a98-ac7a-910a12c8462b","Type":"ContainerStarted","Data":"6a5c25cbae1d76daec5d3f664d655d74b0f29c84616493afb5c6495dc00fefc4"} Oct 30 00:48:20.527174 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:20.527144 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:20.532076 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:20.532054 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:20.577608 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:20.577563 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" podStartSLOduration=1.817647425 podStartE2EDuration="4.57754894s" totalImagesPullingTime="2.759901515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:20.541134101 +0000 UTC m=+291.383694694" watchObservedRunningTime="2026-10-30 00:48:20.57754894 +0000 UTC m=+291.420109590" Oct 30 00:48:21.492948 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:21.492907 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:21.493936 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:21.493918 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g" Oct 30 00:48:25.426787 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:48:25.426754 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:48:29.305425 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:29.305400 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:48:29.306603 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:29.306579 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:48:29.485191 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:48:29.485163 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-d777n" Oct 30 00:48:55.426402 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:48:55.426368 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:49:14.604908 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.604854 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-96qsm"] Oct 30 00:49:14.608411 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.608389 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.630643 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.630618 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 00:49:14.630806 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.630759 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f9m5r\"" Oct 30 00:49:14.632350 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.632330 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-96qsm"] Oct 30 00:49:14.639099 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.639077 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c64d48-a509-44d7-b54c-136d59eedcd9-cert\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.639204 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.639119 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m67v2\" (UniqueName: \"kubernetes.io/projected/f8c64d48-a509-44d7-b54c-136d59eedcd9-kube-api-access-m67v2\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.650487 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.650464 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:49:14.653812 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.653796 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.667958 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.667938 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-htltf\"" Oct 30 00:49:14.668058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.668010 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 00:49:14.673414 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.673397 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:49:14.691577 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.691558 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hqgk7"] Oct 30 00:49:14.695188 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.695170 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.722080 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.722054 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hqgk7"] Oct 30 00:49:14.739600 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739578 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b7k\" (UniqueName: \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.739745 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739627 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c64d48-a509-44d7-b54c-136d59eedcd9-cert\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.739745 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739673 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.739745 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739696 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbxh\" (UniqueName: \"kubernetes.io/projected/316952b3-df35-4305-8b20-0b5aa60957e7-kube-api-access-lfbxh\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.739745 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739723 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316952b3-df35-4305-8b20-0b5aa60957e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.739998 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.739838 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m67v2\" (UniqueName: \"kubernetes.io/projected/f8c64d48-a509-44d7-b54c-136d59eedcd9-kube-api-access-m67v2\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.742310 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.742291 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8c64d48-a509-44d7-b54c-136d59eedcd9-cert\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.758661 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.758637 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m67v2\" (UniqueName: \"kubernetes.io/projected/f8c64d48-a509-44d7-b54c-136d59eedcd9-kube-api-access-m67v2\") pod \"kserve-controller-manager-776cc66d47-96qsm\" (UID: \"f8c64d48-a509-44d7-b54c-136d59eedcd9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.772774 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.772732 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-bjbg9"] Oct 30 00:49:14.772941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.772824 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4vc2m\"" Oct 30 00:49:14.773017 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.772960 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 00:49:14.776416 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.776398 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.804609 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.804577 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-bjbg9"] Oct 30 00:49:14.840918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.840882 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-metrics-certs\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.841063 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841008 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b7k\" (UniqueName: \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.841063 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841051 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvt2v\" (UniqueName: \"kubernetes.io/projected/626858bc-fea2-4656-9704-0684969f1490-kube-api-access-dvt2v\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.841150 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841089 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-cert\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.841150 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841117 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.841312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841297 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbxh\" (UniqueName: \"kubernetes.io/projected/316952b3-df35-4305-8b20-0b5aa60957e7-kube-api-access-lfbxh\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.841400 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.841354 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316952b3-df35-4305-8b20-0b5aa60957e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.843723 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.843697 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.843850 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.843728 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/316952b3-df35-4305-8b20-0b5aa60957e7-tls-certs\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.858064 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.857993 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbxh\" (UniqueName: \"kubernetes.io/projected/316952b3-df35-4305-8b20-0b5aa60957e7-kube-api-access-lfbxh\") pod \"model-serving-api-75bd89fdfc-hqgk7\" (UID: \"316952b3-df35-4305-8b20-0b5aa60957e7\") " pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:14.858181 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.858084 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b7k\" (UniqueName: \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\") pod \"llmisvc-controller-manager-869d5dcc5d-fngd9\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:14.862634 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.862618 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 00:49:14.862723 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.862707 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qrrwb\"" Oct 30 00:49:14.862819 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.862808 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 00:49:14.920999 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.920978 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:14.941872 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.941839 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvt2v\" (UniqueName: \"kubernetes.io/projected/626858bc-fea2-4656-9704-0684969f1490-kube-api-access-dvt2v\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.941976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.941900 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-cert\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.941976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.941949 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-metrics-certs\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.944574 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.944552 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-cert\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.944670 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.944573 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/626858bc-fea2-4656-9704-0684969f1490-metrics-certs\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.958303 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.958279 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvt2v\" (UniqueName: \"kubernetes.io/projected/626858bc-fea2-4656-9704-0684969f1490-kube-api-access-dvt2v\") pod \"odh-model-controller-f87fcf56c-bjbg9\" (UID: \"626858bc-fea2-4656-9704-0684969f1490\") " pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:14.965155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:14.965134 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:15.008220 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.008191 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:15.089838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.089809 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:15.214722 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.214688 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-96qsm"] Oct 30 00:49:15.217662 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:49:15.217639 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c64d48_a509_44d7_b54c_136d59eedcd9.slice/crio-6993da447835e19215f1d0d0643d2798cb9c303d76ced24264f7f6f7f96a7ad7 WatchSource:0}: Error finding container 6993da447835e19215f1d0d0643d2798cb9c303d76ced24264f7f6f7f96a7ad7: Status 404 returned error can't find the container with id 6993da447835e19215f1d0d0643d2798cb9c303d76ced24264f7f6f7f96a7ad7 Oct 30 00:49:15.218979 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.218963 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:49:15.263999 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.263963 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:49:15.266797 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:49:15.266762 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c281605_cd19_4f79_9fc9_a7e1bbef3f6e.slice/crio-6b09c0693bbea6ba40e61f008c6318f6e0e924170682b4a7f3dec2b27de38e76 WatchSource:0}: Error finding container 6b09c0693bbea6ba40e61f008c6318f6e0e924170682b4a7f3dec2b27de38e76: Status 404 returned error can't find the container with id 6b09c0693bbea6ba40e61f008c6318f6e0e924170682b4a7f3dec2b27de38e76 Oct 30 00:49:15.315731 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.315658 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hqgk7"] Oct 30 00:49:15.320995 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:49:15.320963 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316952b3_df35_4305_8b20_0b5aa60957e7.slice/crio-0b1b0ac500004ac84ef8be6359eecb839a570e9503d00c1c8a0ea28519d7841b WatchSource:0}: Error finding container 0b1b0ac500004ac84ef8be6359eecb839a570e9503d00c1c8a0ea28519d7841b: Status 404 returned error can't find the container with id 0b1b0ac500004ac84ef8be6359eecb839a570e9503d00c1c8a0ea28519d7841b Oct 30 00:49:15.391585 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.391490 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-bjbg9"] Oct 30 00:49:15.399922 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:49:15.399888 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626858bc_fea2_4656_9704_0684969f1490.slice/crio-989e2cbb03f5b44ae800e39518c7c94366f9ece0c4be40eccc543dc4e05ef0ab WatchSource:0}: Error finding container 989e2cbb03f5b44ae800e39518c7c94366f9ece0c4be40eccc543dc4e05ef0ab: Status 404 returned error can't find the container with id 989e2cbb03f5b44ae800e39518c7c94366f9ece0c4be40eccc543dc4e05ef0ab Oct 30 00:49:15.751829 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.751741 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" event={"ID":"626858bc-fea2-4656-9704-0684969f1490","Type":"ContainerStarted","Data":"989e2cbb03f5b44ae800e39518c7c94366f9ece0c4be40eccc543dc4e05ef0ab"} Oct 30 00:49:15.756383 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.756347 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" event={"ID":"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e","Type":"ContainerStarted","Data":"6b09c0693bbea6ba40e61f008c6318f6e0e924170682b4a7f3dec2b27de38e76"} Oct 30 00:49:15.769010 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.761908 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" event={"ID":"f8c64d48-a509-44d7-b54c-136d59eedcd9","Type":"ContainerStarted","Data":"6993da447835e19215f1d0d0643d2798cb9c303d76ced24264f7f6f7f96a7ad7"} Oct 30 00:49:15.769010 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:15.765092 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" event={"ID":"316952b3-df35-4305-8b20-0b5aa60957e7","Type":"ContainerStarted","Data":"0b1b0ac500004ac84ef8be6359eecb839a570e9503d00c1c8a0ea28519d7841b"} Oct 30 00:49:18.795518 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:18.795421 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" event={"ID":"316952b3-df35-4305-8b20-0b5aa60957e7","Type":"ContainerStarted","Data":"3de52e436f360e59d2cf629a83d515e895876841f84cdbfeabf88b3945d05844"} Oct 30 00:49:18.796011 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:18.795555 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:18.828880 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:18.828806 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" podStartSLOduration=1.7526152860000002 podStartE2EDuration="4.82879074s" totalImagesPullingTime="3.076175454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:18.827614441 +0000 UTC m=+349.670175061" watchObservedRunningTime="2026-10-30 00:49:18.82879074 +0000 UTC m=+349.671351335" Oct 30 00:49:19.801700 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.801662 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" event={"ID":"626858bc-fea2-4656-9704-0684969f1490","Type":"ContainerStarted","Data":"054f77ef42be2b77d9e8e5765e3be9f22be42400ccb23848632427bc08066386"} Oct 30 00:49:19.802199 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.801797 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:19.803526 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.803499 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" event={"ID":"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e","Type":"ContainerStarted","Data":"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5"} Oct 30 00:49:19.803671 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.803600 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:19.804982 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.804958 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" event={"ID":"f8c64d48-a509-44d7-b54c-136d59eedcd9","Type":"ContainerStarted","Data":"0c26d06204701883428aa8920dbbb47ec47684c2db70e5920ba828635cac88f3"} Oct 30 00:49:19.805126 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.805111 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:19.835092 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.835018 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" podStartSLOduration=1.789500924 podStartE2EDuration="5.835006761s" totalImagesPullingTime="4.045505837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:19.8346696 +0000 UTC m=+350.677230193" watchObservedRunningTime="2026-10-30 00:49:19.835006761 +0000 UTC m=+350.677567390" Oct 30 00:49:19.868682 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.868622 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" podStartSLOduration=1.772974285 podStartE2EDuration="5.86860691s" totalImagesPullingTime="4.095632625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:19.867824347 +0000 UTC m=+350.710384942" watchObservedRunningTime="2026-10-30 00:49:19.86860691 +0000 UTC m=+350.711167506" Oct 30 00:49:19.900596 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:19.900544 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" podStartSLOduration=1.68220725 podStartE2EDuration="5.900531035s" totalImagesPullingTime="4.218323785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:19.900215 +0000 UTC m=+350.742775593" watchObservedRunningTime="2026-10-30 00:49:19.900531035 +0000 UTC m=+350.743091619" Oct 30 00:49:25.428923 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:49:25.428879 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:49:29.809126 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:29.809100 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-hqgk7" Oct 30 00:49:30.813136 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:30.813106 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-bjbg9" Oct 30 00:49:50.812550 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:50.812471 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:49:50.815523 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:49:50.815502 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-96qsm" Oct 30 00:49:55.425767 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:49:55.425722 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:50:05.369852 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.369807 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj"] Oct 30 00:50:05.378459 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.378437 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.392408 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.392378 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj"] Oct 30 00:50:05.400272 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.400238 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 00:50:05.400272 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.400249 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 00:50:05.400455 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.400321 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-nzkjb\"" Oct 30 00:50:05.464834 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.464800 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq"] Oct 30 00:50:05.467795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.467773 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.482227 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.482205 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jcrzq\"" Oct 30 00:50:05.486825 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.486802 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq"] Oct 30 00:50:05.494634 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.494605 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.494783 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.494666 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.494852 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.494783 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.494852 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.494827 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fjr\" (UniqueName: \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.494985 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.494964 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbrc\" (UniqueName: \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.495063 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.495045 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.565978 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.565943 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2"] Oct 30 00:50:05.569139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.569119 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.581820 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.581802 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lltdx\"" Oct 30 00:50:05.587908 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.587887 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2"] Oct 30 00:50:05.595810 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.595791 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbrc\" (UniqueName: \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.595934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.595842 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.595934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.595903 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.595934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.595927 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.596123 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.595954 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwgv\" (UniqueName: \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.596123 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596030 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.596123 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596071 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.596123 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596109 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.596334 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596149 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fjr\" (UniqueName: \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.596392 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596332 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.596392 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596352 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.596392 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596372 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.596525 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.596450 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.612606 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.612570 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbrc\" (UniqueName: \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.612740 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.612720 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fjr\" (UniqueName: \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.664698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.664666 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx"] Oct 30 00:50:05.667702 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.667684 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.679724 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.679708 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-pz84t\"" Oct 30 00:50:05.689080 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.689058 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx"] Oct 30 00:50:05.691409 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.691387 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:05.697449 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697427 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.697539 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697467 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.697592 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697532 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwgv\" (UniqueName: \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.697695 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697677 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.697755 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697712 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjdp\" (UniqueName: \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.697826 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697800 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.697826 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.697803 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.698112 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.698090 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.714182 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.714159 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwgv\" (UniqueName: \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.781337 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.781305 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:05.799112 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.799078 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.799279 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.799137 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.799279 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.799169 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjdp\" (UniqueName: \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.799473 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.799455 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.799538 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.799503 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.820695 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.820661 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjdp\" (UniqueName: \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.881926 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.881879 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:05.981095 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.981069 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:05.986486 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:05.986458 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj"] Oct 30 00:50:05.990999 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:50:05.990972 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a2f1c9_9b5a_47d4_8ec6_30dcc0c63461.slice/crio-b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3 WatchSource:0}: Error finding container b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3: Status 404 returned error can't find the container with id b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3 Oct 30 00:50:06.022361 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:06.022317 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" event={"ID":"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461","Type":"ContainerStarted","Data":"b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3"} Oct 30 00:50:06.075690 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:06.075657 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq"] Oct 30 00:50:06.078477 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:50:06.078453 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf860c552_c742_4bb0_854a_3ae9bea557de.slice/crio-4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc WatchSource:0}: Error finding container 4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc: Status 404 returned error can't find the container with id 4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc Oct 30 00:50:06.173755 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:06.173729 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2"] Oct 30 00:50:06.175980 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:50:06.175952 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8797f5_d340_4c51_87b8_d32bf904528a.slice/crio-6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514 WatchSource:0}: Error finding container 6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514: Status 404 returned error can't find the container with id 6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514 Oct 30 00:50:06.279294 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:06.279265 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx"] Oct 30 00:50:06.336679 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:50:06.336646 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d6377d_3300_4c47_a942_ff78f0a09a0d.slice/crio-873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb WatchSource:0}: Error finding container 873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb: Status 404 returned error can't find the container with id 873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb Oct 30 00:50:07.028128 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.028087 5548 generic.go:356] "Generic (PLEG): container finished" podID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerID="814c0b319665107b91ba9689443f0d0eb6509962d170cebbf4634a2f36e80902" exitCode=0 Oct 30 00:50:07.028562 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.028179 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" event={"ID":"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461","Type":"ContainerDied","Data":"814c0b319665107b91ba9689443f0d0eb6509962d170cebbf4634a2f36e80902"} Oct 30 00:50:07.029807 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.029776 5548 generic.go:356] "Generic (PLEG): container finished" podID="f860c552-c742-4bb0-854a-3ae9bea557de" containerID="b9e69afc35b8659e73d7b04cafc174f9685d0a731feb66715d79d422537cd9c3" exitCode=0 Oct 30 00:50:07.029936 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.029886 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" event={"ID":"f860c552-c742-4bb0-854a-3ae9bea557de","Type":"ContainerDied","Data":"b9e69afc35b8659e73d7b04cafc174f9685d0a731feb66715d79d422537cd9c3"} Oct 30 00:50:07.029936 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.029929 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" event={"ID":"f860c552-c742-4bb0-854a-3ae9bea557de","Type":"ContainerStarted","Data":"4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc"} Oct 30 00:50:07.031561 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.031538 5548 generic.go:356] "Generic (PLEG): container finished" podID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerID="c9a811db8e9e3b4b5d64a6440ebeb9b17d93ce9fb4bec0866b344cab58b0751b" exitCode=0 Oct 30 00:50:07.031668 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.031564 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" event={"ID":"47d6377d-3300-4c47-a942-ff78f0a09a0d","Type":"ContainerDied","Data":"c9a811db8e9e3b4b5d64a6440ebeb9b17d93ce9fb4bec0866b344cab58b0751b"} Oct 30 00:50:07.031668 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.031604 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" event={"ID":"47d6377d-3300-4c47-a942-ff78f0a09a0d","Type":"ContainerStarted","Data":"873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb"} Oct 30 00:50:07.033159 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.033138 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerID="480b4ec1d207b98bbb2e703cf820a8ac2bb453d6c2e88e0f2156d587b24809a0" exitCode=0 Oct 30 00:50:07.033238 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.033163 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" event={"ID":"4e8797f5-d340-4c51-87b8-d32bf904528a","Type":"ContainerDied","Data":"480b4ec1d207b98bbb2e703cf820a8ac2bb453d6c2e88e0f2156d587b24809a0"} Oct 30 00:50:07.033238 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:07.033195 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" event={"ID":"4e8797f5-d340-4c51-87b8-d32bf904528a","Type":"ContainerStarted","Data":"6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514"} Oct 30 00:50:08.039924 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:08.039899 5548 generic.go:356] "Generic (PLEG): container finished" podID="f860c552-c742-4bb0-854a-3ae9bea557de" containerID="23700aadf9a613abfe2675c961d355fd4767f54e3d7476d5bdf4b883f981954d" exitCode=0 Oct 30 00:50:08.040308 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:08.039980 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" event={"ID":"f860c552-c742-4bb0-854a-3ae9bea557de","Type":"ContainerDied","Data":"23700aadf9a613abfe2675c961d355fd4767f54e3d7476d5bdf4b883f981954d"} Oct 30 00:50:08.042147 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:08.042118 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerID="9e6223395b7cb6ca594fc57651804438eec46d565888cab9572964895538eda2" exitCode=0 Oct 30 00:50:08.042273 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:08.042160 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" event={"ID":"4e8797f5-d340-4c51-87b8-d32bf904528a","Type":"ContainerDied","Data":"9e6223395b7cb6ca594fc57651804438eec46d565888cab9572964895538eda2"} Oct 30 00:50:09.048687 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.048602 5548 generic.go:356] "Generic (PLEG): container finished" podID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerID="033c5ca2f2d7a8a0f4d74ed53ecbb66cc04fa3e05001901086d679eba79e218b" exitCode=0 Oct 30 00:50:09.049139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.048686 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" event={"ID":"47d6377d-3300-4c47-a942-ff78f0a09a0d","Type":"ContainerDied","Data":"033c5ca2f2d7a8a0f4d74ed53ecbb66cc04fa3e05001901086d679eba79e218b"} Oct 30 00:50:09.051198 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.051173 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerID="7455c1a3eeff38ee88817752bf14503ca8de5fb4ac20e9c425ac91be9ab07752" exitCode=0 Oct 30 00:50:09.051319 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.051261 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" event={"ID":"4e8797f5-d340-4c51-87b8-d32bf904528a","Type":"ContainerDied","Data":"7455c1a3eeff38ee88817752bf14503ca8de5fb4ac20e9c425ac91be9ab07752"} Oct 30 00:50:09.053371 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.053348 5548 generic.go:356] "Generic (PLEG): container finished" podID="f860c552-c742-4bb0-854a-3ae9bea557de" containerID="923105e14f5acaf790ff01e7eb10769381ffb2aea49e67dc925e2e3c9253b8d6" exitCode=0 Oct 30 00:50:09.053461 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:09.053402 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" event={"ID":"f860c552-c742-4bb0-854a-3ae9bea557de","Type":"ContainerDied","Data":"923105e14f5acaf790ff01e7eb10769381ffb2aea49e67dc925e2e3c9253b8d6"} Oct 30 00:50:10.060219 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.060181 5548 generic.go:356] "Generic (PLEG): container finished" podID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerID="5b8ee8bfa5d5586bde97e3ee9349c9b874472da5b4523aca3c9be1e3f634917c" exitCode=0 Oct 30 00:50:10.060707 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.060272 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" event={"ID":"47d6377d-3300-4c47-a942-ff78f0a09a0d","Type":"ContainerDied","Data":"5b8ee8bfa5d5586bde97e3ee9349c9b874472da5b4523aca3c9be1e3f634917c"} Oct 30 00:50:10.191904 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.191880 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:10.223590 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.223568 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:10.237839 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.237816 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\") pod \"4e8797f5-d340-4c51-87b8-d32bf904528a\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " Oct 30 00:50:10.237961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.237888 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\" (UniqueName: \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\") pod \"4e8797f5-d340-4c51-87b8-d32bf904528a\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " Oct 30 00:50:10.237961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.237926 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\") pod \"4e8797f5-d340-4c51-87b8-d32bf904528a\" (UID: \"4e8797f5-d340-4c51-87b8-d32bf904528a\") " Oct 30 00:50:10.238490 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.238462 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle" pod "4e8797f5-d340-4c51-87b8-d32bf904528a" (UID: "4e8797f5-d340-4c51-87b8-d32bf904528a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:10.240135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.240107 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv" pod "4e8797f5-d340-4c51-87b8-d32bf904528a" (UID: "4e8797f5-d340-4c51-87b8-d32bf904528a"). InnerVolumeSpecName "kube-api-access-7dwgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:10.246635 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.246614 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util" pod "4e8797f5-d340-4c51-87b8-d32bf904528a" (UID: "4e8797f5-d340-4c51-87b8-d32bf904528a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:10.339451 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339371 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\") pod \"f860c552-c742-4bb0-854a-3ae9bea557de\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " Oct 30 00:50:10.339451 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339446 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\" (UniqueName: \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\") pod \"f860c552-c742-4bb0-854a-3ae9bea557de\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " Oct 30 00:50:10.339672 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339549 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\") pod \"f860c552-c742-4bb0-854a-3ae9bea557de\" (UID: \"f860c552-c742-4bb0-854a-3ae9bea557de\") " Oct 30 00:50:10.339838 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339821 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.339923 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339850 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwgv\" (UniqueName: \"kubernetes.io/projected/4e8797f5-d340-4c51-87b8-d32bf904528a-kube-api-access-7dwgv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.339923 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.339893 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e8797f5-d340-4c51-87b8-d32bf904528a-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.340068 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.340041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle" pod "f860c552-c742-4bb0-854a-3ae9bea557de" (UID: "f860c552-c742-4bb0-854a-3ae9bea557de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:10.341686 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.341657 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr" pod "f860c552-c742-4bb0-854a-3ae9bea557de" (UID: "f860c552-c742-4bb0-854a-3ae9bea557de"). InnerVolumeSpecName "kube-api-access-95fjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:10.345000 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.344979 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util" pod "f860c552-c742-4bb0-854a-3ae9bea557de" (UID: "f860c552-c742-4bb0-854a-3ae9bea557de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:10.440822 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.440794 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fjr\" (UniqueName: \"kubernetes.io/projected/f860c552-c742-4bb0-854a-3ae9bea557de-kube-api-access-95fjr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.440822 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.440819 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:10.440822 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:10.440828 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f860c552-c742-4bb0-854a-3ae9bea557de-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:11.069460 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.069428 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" Oct 30 00:50:11.069460 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.069432 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c7fv2" event={"ID":"4e8797f5-d340-4c51-87b8-d32bf904528a","Type":"ContainerDied","Data":"6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514"} Oct 30 00:50:11.069954 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.069530 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e84049a4ac1a57244f3848e153f997525ec086d6c832dc810f2a978146e6514" Oct 30 00:50:11.071464 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.071442 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" Oct 30 00:50:11.071464 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.071453 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4hqq" event={"ID":"f860c552-c742-4bb0-854a-3ae9bea557de","Type":"ContainerDied","Data":"4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc"} Oct 30 00:50:11.071626 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.071487 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da5141cc7347bd327bbacd5ab3990cffd1bb8e17111dee25bd040b9bbedbebc" Oct 30 00:50:11.206609 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.206585 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:11.248790 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.248758 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\") pod \"47d6377d-3300-4c47-a942-ff78f0a09a0d\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " Oct 30 00:50:11.248975 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.248814 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\" (UniqueName: \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\") pod \"47d6377d-3300-4c47-a942-ff78f0a09a0d\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " Oct 30 00:50:11.248975 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.248896 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\") pod \"47d6377d-3300-4c47-a942-ff78f0a09a0d\" (UID: \"47d6377d-3300-4c47-a942-ff78f0a09a0d\") " Oct 30 00:50:11.249752 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.249720 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle" pod "47d6377d-3300-4c47-a942-ff78f0a09a0d" (UID: "47d6377d-3300-4c47-a942-ff78f0a09a0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:11.249898 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.249819 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:11.251261 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.251236 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp" pod "47d6377d-3300-4c47-a942-ff78f0a09a0d" (UID: "47d6377d-3300-4c47-a942-ff78f0a09a0d"). InnerVolumeSpecName "kube-api-access-txjdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:11.255552 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.255522 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util" pod "47d6377d-3300-4c47-a942-ff78f0a09a0d" (UID: "47d6377d-3300-4c47-a942-ff78f0a09a0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:11.350790 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.350715 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d6377d-3300-4c47-a942-ff78f0a09a0d-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:11.350790 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:11.350738 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjdp\" (UniqueName: \"kubernetes.io/projected/47d6377d-3300-4c47-a942-ff78f0a09a0d-kube-api-access-txjdp\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:12.078272 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:12.078244 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" Oct 30 00:50:12.078749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:12.078243 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8rrqx" event={"ID":"47d6377d-3300-4c47-a942-ff78f0a09a0d","Type":"ContainerDied","Data":"873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb"} Oct 30 00:50:12.078749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:12.078368 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="873ddc086113a1125d04da5de07523b29153ad5b20a1ce3a6db11f5d861facfb" Oct 30 00:50:12.080354 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:12.080325 5548 generic.go:356] "Generic (PLEG): container finished" podID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerID="2b42754077b42d43f96ed4c5c952a769be47d27da2fbbc2ff959e9dd61789585" exitCode=0 Oct 30 00:50:12.080500 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:12.080419 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" event={"ID":"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461","Type":"ContainerDied","Data":"2b42754077b42d43f96ed4c5c952a769be47d27da2fbbc2ff959e9dd61789585"} Oct 30 00:50:13.087797 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:13.087760 5548 generic.go:356] "Generic (PLEG): container finished" podID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerID="9aeddbce75c1c32f2b1cbc69079cc017fe7d84bde80d948a4b70e699465a64a5" exitCode=0 Oct 30 00:50:13.088224 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:13.087844 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" event={"ID":"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461","Type":"ContainerDied","Data":"9aeddbce75c1c32f2b1cbc69079cc017fe7d84bde80d948a4b70e699465a64a5"} Oct 30 00:50:14.221938 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.221910 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:14.275365 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.275337 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\") pod \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " Oct 30 00:50:14.275519 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.275376 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\" (UniqueName: \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\") pod \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " Oct 30 00:50:14.275519 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.275435 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\") pod \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\" (UID: \"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461\") " Oct 30 00:50:14.275917 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.275890 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle" pod "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" (UID: "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:14.277779 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.277751 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc" pod "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" (UID: "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461"). InnerVolumeSpecName "kube-api-access-2mbrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:14.283808 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.283776 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util" pod "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" (UID: "40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:14.376711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.376631 5548 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-bundle\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:14.376711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.376662 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mbrc\" (UniqueName: \"kubernetes.io/projected/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-kube-api-access-2mbrc\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:14.376711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:14.376678 5548 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461-util\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:50:15.099334 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:15.099300 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" event={"ID":"40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461","Type":"ContainerDied","Data":"b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3"} Oct 30 00:50:15.099529 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:15.099354 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ee75b6cf453f3591139da1c4f408688fba832a7605d078985d55361159fab3" Oct 30 00:50:15.099529 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:15.099362 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ptjrj" Oct 30 00:50:25.426163 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:50:25.426119 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:50:30.314508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314473 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j"] Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314835 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314846 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314877 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314885 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314898 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314904 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314917 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="pull" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314921 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="pull" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314927 5548 cpu_manager.go:414] "containerMap: removing container" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="pull" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314931 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="pull" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314939 5548 cpu_manager.go:414] "containerMap: removing container" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314943 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="util" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314950 5548 cpu_manager.go:414] "containerMap: removing container" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314954 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314960 5548 cpu_manager.go:414] "containerMap: removing container" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314966 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314980 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314987 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="extract" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.314996 5548 cpu_manager.go:414] "containerMap: removing container" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="pull" Oct 30 00:50:30.314988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315001 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="pull" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315010 5548 cpu_manager.go:414] "containerMap: removing container" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="util" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315016 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="util" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315027 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="pull" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315033 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="pull" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315114 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8797f5-d340-4c51-87b8-d32bf904528a" containerName="extract" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315130 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a2f1c9-9b5a-47d4-8ec6-30dcc0c63461" containerName="extract" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315139 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d6377d-3300-4c47-a942-ff78f0a09a0d" containerName="extract" Oct 30 00:50:30.315875 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.315148 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="f860c552-c742-4bb0-854a-3ae9bea557de" containerName="extract" Oct 30 00:50:30.320057 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.320029 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:30.344983 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.344943 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:50:30.344983 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.344952 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:50:30.345215 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.345113 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dlvxh\"" Oct 30 00:50:30.347141 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.347119 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j"] Oct 30 00:50:30.428579 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.428550 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5bv\" (UniqueName: \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\") pod \"limitador-operator-controller-manager-856dd584c5-bvp9j\" (UID: \"5b930ca1-22c1-4f0f-9e46-5b417fda7af2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:30.529744 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.529714 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5bv\" (UniqueName: \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\") pod \"limitador-operator-controller-manager-856dd584c5-bvp9j\" (UID: \"5b930ca1-22c1-4f0f-9e46-5b417fda7af2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:30.548001 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.547969 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5bv\" (UniqueName: \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\") pod \"limitador-operator-controller-manager-856dd584c5-bvp9j\" (UID: \"5b930ca1-22c1-4f0f-9e46-5b417fda7af2\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:30.636116 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.636039 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:30.932308 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:30.932279 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j"] Oct 30 00:50:30.935033 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:50:30.934994 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b930ca1_22c1_4f0f_9e46_5b417fda7af2.slice/crio-6bb05c63385f2142e31c1334e059e6320523117ddf9a858f969e33b4a24f00e8 WatchSource:0}: Error finding container 6bb05c63385f2142e31c1334e059e6320523117ddf9a858f969e33b4a24f00e8: Status 404 returned error can't find the container with id 6bb05c63385f2142e31c1334e059e6320523117ddf9a858f969e33b4a24f00e8 Oct 30 00:50:31.177777 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:31.177741 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" event={"ID":"5b930ca1-22c1-4f0f-9e46-5b417fda7af2","Type":"ContainerStarted","Data":"6bb05c63385f2142e31c1334e059e6320523117ddf9a858f969e33b4a24f00e8"} Oct 30 00:50:34.195847 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:34.195815 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" event={"ID":"5b930ca1-22c1-4f0f-9e46-5b417fda7af2","Type":"ContainerStarted","Data":"c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17"} Oct 30 00:50:34.196284 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:34.195955 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:34.232642 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:34.232452 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" podStartSLOduration=1.6248633319999999 podStartE2EDuration="4.232436199s" totalImagesPullingTime="2.607572867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:34.23023015 +0000 UTC m=+425.072790745" watchObservedRunningTime="2026-10-30 00:50:34.232436199 +0000 UTC m=+425.074996792" Oct 30 00:50:45.203334 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:50:45.203304 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:50:55.425963 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:50:55.425927 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:51:03.384673 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.384639 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j"] Oct 30 00:51:03.385274 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.384915 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" containerName="manager" containerID="cri-o://c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17" gracePeriod=2 Oct 30 00:51:03.401695 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.401655 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j"] Oct 30 00:51:03.465811 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.465779 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5"] Oct 30 00:51:03.466356 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.466333 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" containerName="manager" Oct 30 00:51:03.466356 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.466356 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" containerName="manager" Oct 30 00:51:03.466546 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.466466 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" containerName="manager" Oct 30 00:51:03.469725 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.469703 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:03.496600 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.496570 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5"] Oct 30 00:51:03.516687 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:03.516644 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-bvp9j\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:03.626509 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.626482 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwf8j\" (UniqueName: \"kubernetes.io/projected/3b77d01f-40c1-4afe-bbf0-ec867396d4d0-kube-api-access-nwf8j\") pod \"limitador-operator-controller-manager-856dd584c5-szlp5\" (UID: \"3b77d01f-40c1-4afe-bbf0-ec867396d4d0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:03.631809 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.631789 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:03.642632 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:03.642608 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-bvp9j\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:03.727542 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.727517 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\" (UniqueName: \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\") pod \"5b930ca1-22c1-4f0f-9e46-5b417fda7af2\" (UID: \"5b930ca1-22c1-4f0f-9e46-5b417fda7af2\") " Oct 30 00:51:03.727730 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.727713 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwf8j\" (UniqueName: \"kubernetes.io/projected/3b77d01f-40c1-4afe-bbf0-ec867396d4d0-kube-api-access-nwf8j\") pod \"limitador-operator-controller-manager-856dd584c5-szlp5\" (UID: \"3b77d01f-40c1-4afe-bbf0-ec867396d4d0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:03.729928 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.729904 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv" pod "5b930ca1-22c1-4f0f-9e46-5b417fda7af2" (UID: "5b930ca1-22c1-4f0f-9e46-5b417fda7af2"). InnerVolumeSpecName "kube-api-access-qv5bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:03.744585 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.744558 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwf8j\" (UniqueName: \"kubernetes.io/projected/3b77d01f-40c1-4afe-bbf0-ec867396d4d0-kube-api-access-nwf8j\") pod \"limitador-operator-controller-manager-856dd584c5-szlp5\" (UID: \"3b77d01f-40c1-4afe-bbf0-ec867396d4d0\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:03.791270 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.791245 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:03.828579 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:03.828546 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5bv\" (UniqueName: \"kubernetes.io/projected/5b930ca1-22c1-4f0f-9e46-5b417fda7af2-kube-api-access-qv5bv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:51:04.088135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.088008 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5"] Oct 30 00:51:04.090508 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:51:04.090474 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b77d01f_40c1_4afe_bbf0_ec867396d4d0.slice/crio-996e77f787cb6064802a2f184dac15d4099f614d7ec28c0f677db3c07d1b3f37 WatchSource:0}: Error finding container 996e77f787cb6064802a2f184dac15d4099f614d7ec28c0f677db3c07d1b3f37: Status 404 returned error can't find the container with id 996e77f787cb6064802a2f184dac15d4099f614d7ec28c0f677db3c07d1b3f37 Oct 30 00:51:04.360164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.360065 5548 generic.go:356] "Generic (PLEG): container finished" podID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" containerID="c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17" exitCode=0 Oct 30 00:51:04.360164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.360121 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:04.360395 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.360168 5548 scope.go:122] "RemoveContainer" containerID="c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17" Oct 30 00:51:04.362220 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.362194 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" event={"ID":"3b77d01f-40c1-4afe-bbf0-ec867396d4d0","Type":"ContainerStarted","Data":"b09e839632f6201ebfc6724d4aefe6feb69490fa4007722f3475e69afe9553dc"} Oct 30 00:51:04.362362 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.362239 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" event={"ID":"3b77d01f-40c1-4afe-bbf0-ec867396d4d0","Type":"ContainerStarted","Data":"996e77f787cb6064802a2f184dac15d4099f614d7ec28c0f677db3c07d1b3f37"} Oct 30 00:51:04.362362 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.362269 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:04.370208 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.370173 5548 scope.go:122] "RemoveContainer" containerID="c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17" Oct 30 00:51:04.370494 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:04.370476 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17\": container with ID starting with c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17 not found: ID does not exist" containerID="c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17" Oct 30 00:51:04.370567 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.370500 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17"} err="failed to get container status \"c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17\": rpc error: code = NotFound desc = could not find container \"c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17\": container with ID starting with c756649c81d6cd468740a0203e8d74260c11feb8d38d95a5d2d71d460457fe17 not found: ID does not exist" Oct 30 00:51:04.371076 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:04.371057 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-bvp9j\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:04.406225 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:04.406180 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" podStartSLOduration=1.406166818 podStartE2EDuration="1.406166818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:04.404928858 +0000 UTC m=+455.247489469" watchObservedRunningTime="2026-10-30 00:51:04.406166818 +0000 UTC m=+455.248727412" Oct 30 00:51:04.415652 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:04.415617 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-bvp9j\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-bvp9j" Oct 30 00:51:05.263955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:05.263923 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b930ca1-22c1-4f0f-9e46-5b417fda7af2" path="/var/lib/kubelet/pods/5b930ca1-22c1-4f0f-9e46-5b417fda7af2/volumes" Oct 30 00:51:15.371448 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:15.371408 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-szlp5" Oct 30 00:51:25.426797 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:25.426755 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:51:36.658404 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.658364 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 00:51:36.662239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.662217 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.676120 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.676065 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5fml\"" Oct 30 00:51:36.688423 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.688395 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 00:51:36.710549 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710521 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710700 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710562 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710700 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710590 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710700 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710669 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710706 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710821 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710838 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710870 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.710899 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.710889 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjllm\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.811720 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.811681 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.811950 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.811745 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.811950 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.811772 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.811950 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.811847 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812200 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812012 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812200 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812062 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812200 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812108 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjllm\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812200 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812175 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812791 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812220 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812791 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812269 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812791 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812483 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812791 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812641 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.812791 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812782 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.813114 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.812903 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.814441 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.814421 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.814650 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.814633 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.830610 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.830575 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.830719 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.830643 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjllm\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\") pod \"maas-default-gateway-openshift-default-67bf986f99-8ntx4\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:36.975147 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:36.975071 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:37.280897 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.280838 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 00:51:37.284211 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:51:37.284187 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e733aae_b7be_48c5_afd1_4fdab74f24a8.slice/crio-dbb14389094d54331cf7c3c0f9d57426326476883af45dd62c73eb64ffd947c9 WatchSource:0}: Error finding container dbb14389094d54331cf7c3c0f9d57426326476883af45dd62c73eb64ffd947c9: Status 404 returned error can't find the container with id dbb14389094d54331cf7c3c0f9d57426326476883af45dd62c73eb64ffd947c9 Oct 30 00:51:37.286263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.286228 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:51:37.286365 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.286318 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:51:37.286431 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.286364 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:51:37.536851 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.536760 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerStarted","Data":"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e"} Oct 30 00:51:37.536851 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.536814 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerStarted","Data":"dbb14389094d54331cf7c3c0f9d57426326476883af45dd62c73eb64ffd947c9"} Oct 30 00:51:37.573929 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.573728 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podStartSLOduration=1.5736685879999999 podStartE2EDuration="1.573668588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:37.572437707 +0000 UTC m=+488.414998300" watchObservedRunningTime="2026-10-30 00:51:37.573668588 +0000 UTC m=+488.416229185" Oct 30 00:51:37.975511 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:37.975476 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:38.980131 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:38.980095 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:39.546869 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:39.546837 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:39.547892 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:51:39.547850 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:51:55.427303 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:51:55.427260 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:52:25.426350 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:52:25.426313 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:52:55.426689 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:52:55.426614 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:53:23.088464 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.088427 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl"] Oct 30 00:53:23.092387 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.092366 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.111473 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.111440 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl"] Oct 30 00:53:23.157879 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.157824 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.158049 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.157899 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnkd\" (UniqueName: \"kubernetes.io/projected/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-kube-api-access-cbnkd\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.258912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.258880 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.258912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.258912 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnkd\" (UniqueName: \"kubernetes.io/projected/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-kube-api-access-cbnkd\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.261606 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.261583 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.275724 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.275694 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnkd\" (UniqueName: \"kubernetes.io/projected/4a80f3cc-cbc5-45fe-92f1-4b89ce732358-kube-api-access-cbnkd\") pod \"llmisvc-controller-manager-6466c554d8-s9xwl\" (UID: \"4a80f3cc-cbc5-45fe-92f1-4b89ce732358\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.405763 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.405726 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:23.704071 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:23.704002 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl"] Oct 30 00:53:23.707168 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:53:23.707139 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a80f3cc_cbc5_45fe_92f1_4b89ce732358.slice/crio-746b03a3969ff1e2358fd908def984e7b86973be48646da447fa30695b6e2d28 WatchSource:0}: Error finding container 746b03a3969ff1e2358fd908def984e7b86973be48646da447fa30695b6e2d28: Status 404 returned error can't find the container with id 746b03a3969ff1e2358fd908def984e7b86973be48646da447fa30695b6e2d28 Oct 30 00:53:24.058532 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:24.058498 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" event={"ID":"4a80f3cc-cbc5-45fe-92f1-4b89ce732358","Type":"ContainerStarted","Data":"746b03a3969ff1e2358fd908def984e7b86973be48646da447fa30695b6e2d28"} Oct 30 00:53:24.547787 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:24.547750 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:24.548228 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:24.547817 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:25.065257 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.065213 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" event={"ID":"4a80f3cc-cbc5-45fe-92f1-4b89ce732358","Type":"ContainerStarted","Data":"4395a318d3e8f22ce0d6d2e18a8e8c348bde7d5ca0a5dfa200e20aab37cdcae9"} Oct 30 00:53:25.065451 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.065335 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:25.066799 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.066777 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerID="224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e" exitCode=0 Oct 30 00:53:25.066934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.066811 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerDied","Data":"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e"} Oct 30 00:53:25.067096 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.067085 5548 scope.go:122] "RemoveContainer" containerID="224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e" Oct 30 00:53:25.068132 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.068102 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:53:25.068232 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.068187 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:53:25.068300 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.068240 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:53:25.099591 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:25.099544 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" podStartSLOduration=1.784894231 podStartE2EDuration="2.099527317s" totalImagesPullingTime="314.633086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:25.097655276 +0000 UTC m=+595.940215869" watchObservedRunningTime="2026-10-30 00:53:25.099527317 +0000 UTC m=+595.942087911" Oct 30 00:53:25.437986 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:53:25.437943 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:53:26.104579 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:26.104544 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerStarted","Data":"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8"} Oct 30 00:53:26.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:26.980253 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:53:26.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:26.980383 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:26.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:26.980426 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:27.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:27.983274 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:27.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:27.983354 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:28.992334 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:28.992296 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:28.992738 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:28.992359 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:29.341928 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:29.341842 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:53:29.344125 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:29.344103 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:53:29.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:29.986341 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:29.986886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:29.986418 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:30.976352 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:30.976309 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8ntx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:53:30.976844 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:30.976389 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:53:31.979730 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:31.979698 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:53:31.980148 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:31.980129 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:53:31.980915 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:31.980891 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 00:53:55.426334 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:53:55.426299 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:53:56.106853 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.106825 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9xwl" Oct 30 00:53:56.170823 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.170794 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:53:56.171082 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.171054 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" podUID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" containerName="manager" containerID="cri-o://7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5" gracePeriod=30 Oct 30 00:53:56.421317 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.421294 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:53:56.553641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.553613 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\" (UniqueName: \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\") pod \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " Oct 30 00:53:56.554037 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.553653 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\" (UniqueName: \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\") pod \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\" (UID: \"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e\") " Oct 30 00:53:56.556077 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.556043 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert" pod "8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" (UID: "8c281605-cd19-4f79-9fc9-a7e1bbef3f6e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:56.556185 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.556099 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k" pod "8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" (UID: "8c281605-cd19-4f79-9fc9-a7e1bbef3f6e"). InnerVolumeSpecName "kube-api-access-86b7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:56.654854 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.654780 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86b7k\" (UniqueName: \"kubernetes.io/projected/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-kube-api-access-86b7k\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:53:56.654854 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:56.654811 5548 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 00:53:57.266153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.266117 5548 generic.go:356] "Generic (PLEG): container finished" podID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" containerID="7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5" exitCode=0 Oct 30 00:53:57.266390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.266164 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" Oct 30 00:53:57.266390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.266158 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" event={"ID":"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e","Type":"ContainerDied","Data":"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5"} Oct 30 00:53:57.266390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.266287 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9" event={"ID":"8c281605-cd19-4f79-9fc9-a7e1bbef3f6e","Type":"ContainerDied","Data":"6b09c0693bbea6ba40e61f008c6318f6e0e924170682b4a7f3dec2b27de38e76"} Oct 30 00:53:57.266390 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.266322 5548 scope.go:122] "RemoveContainer" containerID="7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5" Oct 30 00:53:57.275918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.275901 5548 scope.go:122] "RemoveContainer" containerID="7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5" Oct 30 00:53:57.276153 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:53:57.276137 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5\": container with ID starting with 7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5 not found: ID does not exist" containerID="7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5" Oct 30 00:53:57.276194 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.276161 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5"} err="failed to get container status \"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5\": rpc error: code = NotFound desc = could not find container \"7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5\": container with ID starting with 7995cdd8c0719dd43b7174655994b477f83f5f173e512b01a482888a4ebe93b5 not found: ID does not exist" Oct 30 00:53:57.316518 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.316482 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:53:57.326104 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:57.326083 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-fngd9"] Oct 30 00:53:59.263426 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:53:59.263388 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" path="/var/lib/kubelet/pods/8c281605-cd19-4f79-9fc9-a7e1bbef3f6e/volumes" Oct 30 00:54:25.428691 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:54:25.428643 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:54:55.426451 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:54:55.426409 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:55:07.373710 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.373676 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 00:55:07.374236 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.374061 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" containerName="manager" Oct 30 00:55:07.374236 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.374072 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" containerName="manager" Oct 30 00:55:07.374236 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.374156 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c281605-cd19-4f79-9fc9-a7e1bbef3f6e" containerName="manager" Oct 30 00:55:07.377527 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.377503 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.403108 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.403071 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 00:55:07.479470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479440 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479626 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479481 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479626 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479503 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479626 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479541 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479626 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479619 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479782 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479643 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479782 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479713 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479782 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479773 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.479893 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.479812 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlswk\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581079 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581042 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581079 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581080 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlswk\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581106 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581126 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581151 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581181 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581213 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581238 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581332 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581308 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581642 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581571 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581642 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581590 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581754 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581681 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.581808 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.581790 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.582240 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.582219 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.583669 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.583646 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.583909 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.583887 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.597572 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.597544 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlswk\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.597572 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.597564 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f6dvm\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:07.691249 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:07.691218 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:08.000048 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.000019 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 00:55:08.002325 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:55:08.002299 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e474b8a_3775_4d39_8226_1fef3c8fecde.slice/crio-c2a910db0084446178f869cc69a3abcf7bb7cf5016fceb04107115389098a49c WatchSource:0}: Error finding container c2a910db0084446178f869cc69a3abcf7bb7cf5016fceb04107115389098a49c: Status 404 returned error can't find the container with id c2a910db0084446178f869cc69a3abcf7bb7cf5016fceb04107115389098a49c Oct 30 00:55:08.004382 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.004351 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:08.004453 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.004414 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:08.004453 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.004440 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:08.643455 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.643414 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" event={"ID":"2e474b8a-3775-4d39-8226-1fef3c8fecde","Type":"ContainerStarted","Data":"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695"} Oct 30 00:55:08.643960 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.643472 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" event={"ID":"2e474b8a-3775-4d39-8226-1fef3c8fecde","Type":"ContainerStarted","Data":"c2a910db0084446178f869cc69a3abcf7bb7cf5016fceb04107115389098a49c"} Oct 30 00:55:08.692024 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.691972 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:08.697648 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:08.697604 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podStartSLOduration=1.697590954 podStartE2EDuration="1.697590954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:08.696561729 +0000 UTC m=+699.539122322" watchObservedRunningTime="2026-10-30 00:55:08.697590954 +0000 UTC m=+699.540151616" Oct 30 00:55:09.693729 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:09.693689 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:09.694131 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:09.693755 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:10.692378 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:10.692343 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:10.692575 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:10.692405 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:11.692301 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:11.692261 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:11.692714 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:11.692338 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:12.692205 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:12.692157 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:12.692399 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:12.692210 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:13.692360 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:13.692322 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:13.692550 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:13.692383 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:14.691737 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:14.691700 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:55:14.691966 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:14.691792 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:55:16.244660 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:16.244622 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:16.245159 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:16.245120 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:16.246069 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:16.246050 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 00:55:22.404772 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.404740 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn"] Oct 30 00:55:22.408803 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.408780 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.433979 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.433950 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn"] Oct 30 00:55:22.520553 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520517 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520712 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520560 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vbw\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-kube-api-access-z9vbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520712 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520639 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520712 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520700 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520837 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520726 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520837 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520747 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520837 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520774 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520959 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520906 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.520959 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.520937 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622219 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622183 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622219 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622220 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622242 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622265 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622299 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622317 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622345 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622363 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vbw\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-kube-api-access-z9vbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622680 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622559 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622680 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622659 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622680 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622669 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.622952 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622924 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.623041 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.622979 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.623148 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.623129 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.624843 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.624820 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.624968 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.624951 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.640148 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.640128 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.640580 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.640561 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vbw\" (UniqueName: \"kubernetes.io/projected/9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9-kube-api-access-z9vbw\") pod \"maas-default-gateway-openshift-default-67bf986f99-7r8hn\" (UID: \"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:22.722382 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:22.722307 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:23.033967 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.033893 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn"] Oct 30 00:55:23.037054 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:55:23.037028 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c19f6f0_32e0_4fe0_8e52_0f83a859e6d9.slice/crio-e1130f83c2b3754a9b1fcbf7bf3631e3ff080d30a360f4d43a1e5d8bafb2883e WatchSource:0}: Error finding container e1130f83c2b3754a9b1fcbf7bf3631e3ff080d30a360f4d43a1e5d8bafb2883e: Status 404 returned error can't find the container with id e1130f83c2b3754a9b1fcbf7bf3631e3ff080d30a360f4d43a1e5d8bafb2883e Oct 30 00:55:23.039175 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.039142 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:23.039253 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.039208 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:23.039253 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.039238 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:55:23.719433 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.719398 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" event={"ID":"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9","Type":"ContainerStarted","Data":"a1d7a6847b73ea2ebc9f8c0b8ef29c2b4debca387ed3ca44223b05befe0a8b29"} Oct 30 00:55:23.719834 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.719453 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" event={"ID":"9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9","Type":"ContainerStarted","Data":"e1130f83c2b3754a9b1fcbf7bf3631e3ff080d30a360f4d43a1e5d8bafb2883e"} Oct 30 00:55:23.722725 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.722695 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:23.778884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:23.767112 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podStartSLOduration=1.767093604 podStartE2EDuration="1.767093604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:23.764973655 +0000 UTC m=+714.607534247" watchObservedRunningTime="2026-10-30 00:55:23.767093604 +0000 UTC m=+714.609654211" Oct 30 00:55:24.724323 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:24.724287 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:24.724685 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:24.724340 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:25.426526 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:55:25.426488 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:55:25.723021 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:25.722936 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:25.723021 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:25.722994 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:26.723588 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:26.723553 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:26.724032 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:26.723619 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:27.723374 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:27.723340 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:27.723558 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:27.723389 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:28.723344 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:28.723309 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:28.723734 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:28.723360 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:29.722927 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:29.722888 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:55:29.723139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:29.722948 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:55:31.324386 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:31.324353 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:31.324904 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:31.324733 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:31.325517 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:55:31.325497 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" Oct 30 00:55:55.426328 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:55:55.426242 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:56:00.824824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.824789 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 00:56:00.828438 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.828417 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.841699 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.841677 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-dca38d-serving-cert\"" Oct 30 00:56:00.851594 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.851573 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:56:00.851764 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.851739 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\"" Oct 30 00:56:00.851889 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.851672 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:56:00.851889 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.851816 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:56:00.854801 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.854778 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 00:56:00.873903 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.873853 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.874056 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.873932 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.874056 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.873992 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmxv\" (UniqueName: \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.874056 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.874041 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.975034 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.975004 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.975230 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.975091 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.975230 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.975117 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.975230 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.975145 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmxv\" (UniqueName: \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.975781 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.975756 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.977789 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.977769 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.977984 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.977963 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:00.991961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:00.991936 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmxv\" (UniqueName: \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\") pod \"maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:01.141252 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:01.141153 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:01.469404 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:01.469368 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 00:56:01.475552 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:01.475516 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35f0cf9_21d8_4272_91fd_6acfc5235e42.slice/crio-051c95b6280d101c0d467cbf825f62cec055a7b7dac3bca58d819039a6ab507a WatchSource:0}: Error finding container 051c95b6280d101c0d467cbf825f62cec055a7b7dac3bca58d819039a6ab507a: Status 404 returned error can't find the container with id 051c95b6280d101c0d467cbf825f62cec055a7b7dac3bca58d819039a6ab507a Oct 30 00:56:01.477583 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:01.477563 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:56:01.923478 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:01.923439 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" event={"ID":"d35f0cf9-21d8-4272-91fd-6acfc5235e42","Type":"ContainerStarted","Data":"051c95b6280d101c0d467cbf825f62cec055a7b7dac3bca58d819039a6ab507a"} Oct 30 00:56:02.834707 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.834657 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 00:56:02.844034 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.844008 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.857731 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.857704 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 00:56:02.865679 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.865657 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-c4847c-serving-cert\"" Oct 30 00:56:02.894116 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.894078 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.894275 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.894204 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.894275 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.894238 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.894275 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.894267 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ssq\" (UniqueName: \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.995577 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.995543 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.995577 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.995580 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.996177 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.995597 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ssq\" (UniqueName: \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.996177 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:56:02.995712 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-serving-cert: secret "maas-api-e2e-worker-w6-c4847c-serving-cert" not found Oct 30 00:56:02.996177 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:56:02.995797 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls podName:8e877483-0b31-478b-9be6-b7fe52342ded nodeName:}" failed. No retries permitted until 2026-10-30 00:56:03.495770897 +0000 UTC m=+754.338331483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls") pod "maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" (UID: "8e877483-0b31-478b-9be6-b7fe52342ded") : secret "maas-api-e2e-worker-w6-c4847c-serving-cert" not found Oct 30 00:56:02.996177 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.995716 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.996640 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.996617 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:02.998437 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:02.998414 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:03.013344 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.013317 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ssq\" (UniqueName: \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:03.502786 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.502746 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:03.505750 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.505726 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") pod \"maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:03.760092 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.760007 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:03.936489 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.936450 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" event={"ID":"d35f0cf9-21d8-4272-91fd-6acfc5235e42","Type":"ContainerStarted","Data":"2ba8b11b114ed69cd6d67ec7ddfb8288682a4f9fa5fbed085275588eea4648b6"} Oct 30 00:56:03.936658 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.936546 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:03.974139 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:03.974076 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" podStartSLOduration=1.736255071 podStartE2EDuration="3.974057157s" totalImagesPullingTime="2.237802086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:03.972169486 +0000 UTC m=+754.814730079" watchObservedRunningTime="2026-10-30 00:56:03.974057157 +0000 UTC m=+754.816617751" Oct 30 00:56:04.065786 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:04.065710 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 00:56:04.069669 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:04.069641 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e877483_0b31_478b_9be6_b7fe52342ded.slice/crio-535ff5eb51194c68857e6cbe7fc74b46068ec6ae28424e82d1fdb268c8bf0662 WatchSource:0}: Error finding container 535ff5eb51194c68857e6cbe7fc74b46068ec6ae28424e82d1fdb268c8bf0662: Status 404 returned error can't find the container with id 535ff5eb51194c68857e6cbe7fc74b46068ec6ae28424e82d1fdb268c8bf0662 Oct 30 00:56:04.944297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:04.944256 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" event={"ID":"8e877483-0b31-478b-9be6-b7fe52342ded","Type":"ContainerStarted","Data":"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8"} Oct 30 00:56:04.944485 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:04.944322 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" event={"ID":"8e877483-0b31-478b-9be6-b7fe52342ded","Type":"ContainerStarted","Data":"535ff5eb51194c68857e6cbe7fc74b46068ec6ae28424e82d1fdb268c8bf0662"} Oct 30 00:56:04.944485 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:04.944435 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:04.979571 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:04.979506 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" podStartSLOduration=2.597098695 podStartE2EDuration="2.979490484s" totalImagesPullingTime="382.391789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:04.978570189 +0000 UTC m=+755.821130783" watchObservedRunningTime="2026-10-30 00:56:04.979490484 +0000 UTC m=+755.822051081" Oct 30 00:56:09.950202 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:09.950174 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 00:56:10.128951 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.128914 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 00:56:10.133596 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.133569 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.147645 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.147617 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-82b647-serving-cert\"" Oct 30 00:56:10.149930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.149842 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 00:56:10.270934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.270829 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.270934 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.270890 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.271130 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.270984 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.271173 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.271158 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92pq\" (UniqueName: \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.372263 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.372232 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.372454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.372272 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.372454 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.372301 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.372703 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.372637 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92pq\" (UniqueName: \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.373290 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.373248 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.375111 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.375084 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.375288 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.375271 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.389329 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.389302 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92pq\" (UniqueName: \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\") pod \"maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.450295 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.450264 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:10.746760 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.746615 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 00:56:10.749076 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:10.749045 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06ebece_f45f_4af3_a865_5ff7f6a963ae.slice/crio-81652ff5a0c36b118a11e016869c6518062a4a93f42a6b6742fb9034b795bd75 WatchSource:0}: Error finding container 81652ff5a0c36b118a11e016869c6518062a4a93f42a6b6742fb9034b795bd75: Status 404 returned error can't find the container with id 81652ff5a0c36b118a11e016869c6518062a4a93f42a6b6742fb9034b795bd75 Oct 30 00:56:10.955134 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.955095 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 00:56:10.975878 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:10.975817 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" event={"ID":"f06ebece-f45f-4af3-a865-5ff7f6a963ae","Type":"ContainerStarted","Data":"81652ff5a0c36b118a11e016869c6518062a4a93f42a6b6742fb9034b795bd75"} Oct 30 00:56:11.983956 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:11.983912 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" event={"ID":"f06ebece-f45f-4af3-a865-5ff7f6a963ae","Type":"ContainerStarted","Data":"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7"} Oct 30 00:56:11.984368 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:11.984060 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:12.019529 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:12.019468 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" podStartSLOduration=1.673278978 podStartE2EDuration="2.019451407s" totalImagesPullingTime="346.172429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:12.018627584 +0000 UTC m=+762.861188178" watchObservedRunningTime="2026-10-30 00:56:12.019451407 +0000 UTC m=+762.862012041" Oct 30 00:56:17.014574 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.014537 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 00:56:17.018835 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.018809 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.031309 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.031273 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"openshift-service-ca.crt\"" Oct 30 00:56:17.041573 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.041536 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"kube-root-ca.crt\"" Oct 30 00:56:17.041573 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.041573 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"default-dockercfg-2kgzn\"" Oct 30 00:56:17.041801 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.041677 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"simulator-w1-c104b8-kserve-self-signed-certs\"" Oct 30 00:56:17.043761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.043735 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 00:56:17.144471 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144437 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.144471 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144477 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4p7q\" (UniqueName: \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.144737 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144582 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.144737 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144658 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.144850 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144765 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.144850 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.144840 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.245947 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.245913 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.245982 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246046 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246066 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246098 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4p7q\" (UniqueName: \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246158 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246515 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246499 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.246674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.246661 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.247244 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.247216 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.249011 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.248983 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.249883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.249835 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.263668 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.263638 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4p7q\" (UniqueName: \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\") pod \"simulator-w1-c104b8-kserve-6b4b66784b-dw8pj\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.334767 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.334671 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:17.635194 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.634879 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 00:56:17.636799 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:17.636767 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48654eb2_d22d_444f_b49a_e2f01c69d499.slice/crio-a1e914426e4dbd21714ce996272ff80294d0d068afaaf6e06cabda65f2d05929 WatchSource:0}: Error finding container a1e914426e4dbd21714ce996272ff80294d0d068afaaf6e06cabda65f2d05929: Status 404 returned error can't find the container with id a1e914426e4dbd21714ce996272ff80294d0d068afaaf6e06cabda65f2d05929 Oct 30 00:56:17.998448 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:17.998416 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 00:56:18.024808 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:18.024768 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" event={"ID":"48654eb2-d22d-444f-b49a-e2f01c69d499","Type":"ContainerStarted","Data":"a1e914426e4dbd21714ce996272ff80294d0d068afaaf6e06cabda65f2d05929"} Oct 30 00:56:20.040527 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:20.040491 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" event={"ID":"48654eb2-d22d-444f-b49a-e2f01c69d499","Type":"ContainerStarted","Data":"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5"} Oct 30 00:56:20.077250 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:20.077199 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podStartSLOduration=2.755046012 podStartE2EDuration="4.077185758s" totalImagesPullingTime="1.322139746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:20.076595299 +0000 UTC m=+770.919155895" watchObservedRunningTime="2026-10-30 00:56:20.077185758 +0000 UTC m=+770.919746350" Oct 30 00:56:20.335762 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:20.335676 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:25.427045 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:56:25.427008 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:56:27.335030 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:27.334991 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:27.349485 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:27.349456 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:28.113940 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.113913 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:56:28.929252 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.929222 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 00:56:28.932622 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.932591 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:28.947845 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.947823 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"kube-root-ca.crt\"" Oct 30 00:56:28.955088 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.955058 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"openshift-service-ca.crt\"" Oct 30 00:56:28.955234 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.955102 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"simulator-w0-ba3d7b-kserve-self-signed-certs\"" Oct 30 00:56:28.957534 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.957508 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 00:56:28.958091 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:28.958070 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"default-dockercfg-vh452\"" Oct 30 00:56:29.065460 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065419 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bc8\" (UniqueName: \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.065613 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065497 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.065613 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065566 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.065688 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065624 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.065688 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065653 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.065758 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.065695 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.166604 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.166570 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bc8\" (UniqueName: \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.166908 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.166883 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.167111 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167096 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.167257 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167244 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.167354 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167342 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.167458 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167446 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.167681 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167655 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.168021 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.167895 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.168156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.168133 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.171674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.171157 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.171674 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.171636 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.183806 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.183747 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bc8\" (UniqueName: \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\") pod \"simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.258806 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.258775 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ba3d7b\"/\"default-dockercfg-vh452\"" Oct 30 00:56:29.268297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.268276 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:29.565102 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:29.565080 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 00:56:29.567944 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:29.567912 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd5cd32_44ce_4ab0_8120_49af64d8dc02.slice/crio-d4d425f73f63a64890b15b50e90bada3f71dd02adf2b4c341a5f0f5144d6e36d WatchSource:0}: Error finding container d4d425f73f63a64890b15b50e90bada3f71dd02adf2b4c341a5f0f5144d6e36d: Status 404 returned error can't find the container with id d4d425f73f63a64890b15b50e90bada3f71dd02adf2b4c341a5f0f5144d6e36d Oct 30 00:56:30.117289 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:30.117245 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" event={"ID":"5fd5cd32-44ce-4ab0-8120-49af64d8dc02","Type":"ContainerStarted","Data":"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4"} Oct 30 00:56:30.117289 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:30.117287 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" event={"ID":"5fd5cd32-44ce-4ab0-8120-49af64d8dc02","Type":"ContainerStarted","Data":"d4d425f73f63a64890b15b50e90bada3f71dd02adf2b4c341a5f0f5144d6e36d"} Oct 30 00:56:30.155021 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:30.154971 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podStartSLOduration=2.15495629 podStartE2EDuration="2.15495629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:30.152738778 +0000 UTC m=+780.995299372" watchObservedRunningTime="2026-10-30 00:56:30.15495629 +0000 UTC m=+780.997516879" Oct 30 00:56:30.268693 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:30.268660 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:39.268769 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:39.268732 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:39.281133 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:39.281106 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:40.184872 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:40.184839 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:56:43.388466 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.388428 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 00:56:43.395974 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.395948 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.418623 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.418589 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"kube-root-ca.crt\"" Oct 30 00:56:43.418623 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.418625 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"openshift-service-ca.crt\"" Oct 30 00:56:43.418916 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.418702 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"simulator-w3-82b647-kserve-self-signed-certs\"" Oct 30 00:56:43.418916 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.418840 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"default-dockercfg-g24xb\"" Oct 30 00:56:43.420846 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.420825 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 00:56:43.504830 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.504788 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.505036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.504850 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.505121 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.505024 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5gs\" (UniqueName: \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.505173 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.505115 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.505173 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.505142 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.505323 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.505300 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.605850 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.605814 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.605894 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.605925 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.605953 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5gs\" (UniqueName: \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.605997 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606270 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.606111 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606400 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.606374 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606471 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.606424 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.606471 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.606464 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.608632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.608609 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.608632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.608624 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.624031 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.623975 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5gs\" (UniqueName: \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\") pod \"simulator-w3-82b647-kserve-5f44864fcf-qfpvf\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:43.712339 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:43.712308 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:44.008472 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:44.008450 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 00:56:44.010502 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:44.010474 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6051abfe_1539_41e7_8877_cb21d2e1eef0.slice/crio-9e28bef2dc312ad1e5d91c7cce9b54d53852e3071881a328a0e797ee3cc6e894 WatchSource:0}: Error finding container 9e28bef2dc312ad1e5d91c7cce9b54d53852e3071881a328a0e797ee3cc6e894: Status 404 returned error can't find the container with id 9e28bef2dc312ad1e5d91c7cce9b54d53852e3071881a328a0e797ee3cc6e894 Oct 30 00:56:44.197735 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:44.197700 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" event={"ID":"6051abfe-1539-41e7-8877-cb21d2e1eef0","Type":"ContainerStarted","Data":"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b"} Oct 30 00:56:44.197927 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:44.197745 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" event={"ID":"6051abfe-1539-41e7-8877-cb21d2e1eef0","Type":"ContainerStarted","Data":"9e28bef2dc312ad1e5d91c7cce9b54d53852e3071881a328a0e797ee3cc6e894"} Oct 30 00:56:44.231344 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:44.231294 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podStartSLOduration=1.231281551 podStartE2EDuration="1.231281551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:44.230757664 +0000 UTC m=+795.073318257" watchObservedRunningTime="2026-10-30 00:56:44.231281551 +0000 UTC m=+795.073842145" Oct 30 00:56:44.713149 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:44.713118 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:51.195532 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.195500 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 00:56:51.199410 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.199389 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.221755 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.221727 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"default-dockercfg-5p7xd\"" Oct 30 00:56:51.221922 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.221809 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"simulator-w2-fd3281-kserve-self-signed-certs\"" Oct 30 00:56:51.224372 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.224341 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 00:56:51.249931 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.249902 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"kube-root-ca.crt\"" Oct 30 00:56:51.250051 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.249987 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"openshift-service-ca.crt\"" Oct 30 00:56:51.287508 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287478 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.287669 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287523 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.287669 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287593 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.287669 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287642 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.287795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287690 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.287795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.287750 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tkt\" (UniqueName: \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.388632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.388593 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tkt\" (UniqueName: \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.388941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.388909 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389091 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389000 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389091 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389040 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389091 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389080 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389255 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389210 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389370 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389347 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389694 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389666 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.389822 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.389759 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.392058 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.392038 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.392468 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.392448 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.406796 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.406763 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tkt\" (UniqueName: \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\") pod \"simulator-w2-fd3281-kserve-66486987bf-pfss4\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.516147 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.516054 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:51.816358 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:51.816322 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 00:56:51.817662 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:56:51.817621 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3bf8af7_6f75_4cc1_89d1_64611c87ea4e.slice/crio-28ec40137e2e0a077767e312453461130c1d53b8d4e062499455ddc30c56ecd5 WatchSource:0}: Error finding container 28ec40137e2e0a077767e312453461130c1d53b8d4e062499455ddc30c56ecd5: Status 404 returned error can't find the container with id 28ec40137e2e0a077767e312453461130c1d53b8d4e062499455ddc30c56ecd5 Oct 30 00:56:52.244164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:52.244125 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" event={"ID":"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e","Type":"ContainerStarted","Data":"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df"} Oct 30 00:56:52.244164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:52.244177 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" event={"ID":"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e","Type":"ContainerStarted","Data":"28ec40137e2e0a077767e312453461130c1d53b8d4e062499455ddc30c56ecd5"} Oct 30 00:56:52.278098 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:52.278013 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podStartSLOduration=1.277995855 podStartE2EDuration="1.277995855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:52.276989936 +0000 UTC m=+803.119550533" watchObservedRunningTime="2026-10-30 00:56:52.277995855 +0000 UTC m=+803.120556450" Oct 30 00:56:52.516750 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:52.516673 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:56:53.713008 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:53.712976 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:53.725526 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:53.725501 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:54.267143 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:56:54.267115 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:56:55.426758 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:56:55.426713 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:57:01.516297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:01.516259 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:57:01.529101 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:01.529073 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:57:02.312582 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:02.312555 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:57:14.661076 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.661040 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 00:57:14.664654 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.664635 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.684607 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.684580 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"kube-root-ca.crt\"" Oct 30 00:57:14.684607 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.684586 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"openshift-service-ca.crt\"" Oct 30 00:57:14.684607 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.684603 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"default-dockercfg-tmv96\"" Oct 30 00:57:14.695045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.695019 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"premium-w4-d827b7-kserve-self-signed-certs\"" Oct 30 00:57:14.698540 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.698513 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 00:57:14.710806 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.710780 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.710953 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.710815 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.710953 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.710833 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.711050 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.711034 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.711098 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.711080 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4wm\" (UniqueName: \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.711252 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.711228 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812366 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812320 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812563 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812381 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812563 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812410 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812563 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812514 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812563 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812554 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4wm\" (UniqueName: \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.812753 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.812623 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.813380 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.813147 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.813380 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.813212 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.813380 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.813243 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.815269 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.815246 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.815532 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.815512 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.829123 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.829096 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4wm\" (UniqueName: \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\") pod \"premium-w4-d827b7-kserve-7dc964f56b-rpn97\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:14.976724 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:14.976645 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:15.276488 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:15.276466 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 00:57:15.278524 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:57:15.278492 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd740d6f4_0229_45e3_8b13_ddd190c044bc.slice/crio-b874004160de672aa84d7dfc1ef89745240a5311804537b7d3e91c8e3a49b1cc WatchSource:0}: Error finding container b874004160de672aa84d7dfc1ef89745240a5311804537b7d3e91c8e3a49b1cc: Status 404 returned error can't find the container with id b874004160de672aa84d7dfc1ef89745240a5311804537b7d3e91c8e3a49b1cc Oct 30 00:57:15.376874 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:15.376819 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" event={"ID":"d740d6f4-0229-45e3-8b13-ddd190c044bc","Type":"ContainerStarted","Data":"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15"} Oct 30 00:57:15.376874 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:15.376878 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" event={"ID":"d740d6f4-0229-45e3-8b13-ddd190c044bc","Type":"ContainerStarted","Data":"b874004160de672aa84d7dfc1ef89745240a5311804537b7d3e91c8e3a49b1cc"} Oct 30 00:57:15.418741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:15.414677 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podStartSLOduration=1.4146585329999999 podStartE2EDuration="1.414658533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:15.414044944 +0000 UTC m=+826.256605549" watchObservedRunningTime="2026-10-30 00:57:15.414658533 +0000 UTC m=+826.257219127" Oct 30 00:57:15.977504 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:15.977473 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:24.977567 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:24.977528 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:24.991061 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:24.991034 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:25.428072 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:57:25.428035 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:57:25.444322 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:25.444298 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:57:35.365480 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.365440 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 00:57:35.368970 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.368949 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.382046 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.382024 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dca38d-openshift-default-dockercfg-b9zlp\"" Oct 30 00:57:35.395291 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.395266 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 00:57:35.519149 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519121 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519336 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519175 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519336 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519207 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmq4\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519336 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519316 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519449 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519348 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519449 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519372 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519517 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519491 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519551 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519527 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.519593 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.519555 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.566997 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.566956 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 00:57:35.570129 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.570113 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.586004 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.585970 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c104b8-openshift-default-dockercfg-m9r8p\"" Oct 30 00:57:35.607810 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.607780 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 00:57:35.621155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621089 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621158 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621185 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmq4\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621216 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621261 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621438 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621508 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621622 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621553 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621622 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621584 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621727 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621647 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621727 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621681 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.621896 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.621851 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.622041 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.622020 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.622202 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.622186 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.624085 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.624057 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.624274 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.624257 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.649367 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.649305 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmq4\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.652830 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.652801 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.681576 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.681545 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:35.723652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.723610 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.723832 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.723759 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcs2\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.723964 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.723840 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724024 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.723959 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724065 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.724025 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724117 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.724070 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724166 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.724141 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724217 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.724198 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.724270 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.724241 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.824955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.824922 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.824955 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.824954 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825186 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.824984 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825186 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825022 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825186 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825048 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825186 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825078 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825186 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825120 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825439 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825187 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcs2\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825439 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825242 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825439 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825394 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825654 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825606 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825667 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.825741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.825676 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.826083 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.826061 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.827549 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.827524 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.827708 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.827689 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.841776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.841738 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.842205 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.842182 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcs2\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:35.884235 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:35.884169 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:36.002225 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.002202 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 00:57:36.004467 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:57:36.004438 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccd4ec3_abd3_48fb_a7bf_313016e396a7.slice/crio-eb978d07dc7f8e95543d07661f0af8a22f63fd2deabb57167ec2cc609dc282b9 WatchSource:0}: Error finding container eb978d07dc7f8e95543d07661f0af8a22f63fd2deabb57167ec2cc609dc282b9: Status 404 returned error can't find the container with id eb978d07dc7f8e95543d07661f0af8a22f63fd2deabb57167ec2cc609dc282b9 Oct 30 00:57:36.006698 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.006668 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.006756 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.006733 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.006802 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.006762 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.183234 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.183204 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 00:57:36.186808 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:57:36.186780 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63acbafe_8598_4c0b_a81d_8ba6d0f97799.slice/crio-8372e9f50d22014b4d4a16123c5f84f73fc13e7923a8a4d9002993907144f920 WatchSource:0}: Error finding container 8372e9f50d22014b4d4a16123c5f84f73fc13e7923a8a4d9002993907144f920: Status 404 returned error can't find the container with id 8372e9f50d22014b4d4a16123c5f84f73fc13e7923a8a4d9002993907144f920 Oct 30 00:57:36.189282 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.189253 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.189361 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.189316 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.189361 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.189347 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:36.500727 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.500623 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" event={"ID":"63acbafe-8598-4c0b-a81d-8ba6d0f97799","Type":"ContainerStarted","Data":"a257fbf5b8ecfc368000118c8e99b6142e503719048fab8cc50ff9f89479106e"} Oct 30 00:57:36.500727 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.500694 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" event={"ID":"63acbafe-8598-4c0b-a81d-8ba6d0f97799","Type":"ContainerStarted","Data":"8372e9f50d22014b4d4a16123c5f84f73fc13e7923a8a4d9002993907144f920"} Oct 30 00:57:36.502908 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.502878 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" event={"ID":"0ccd4ec3-abd3-48fb-a7bf-313016e396a7","Type":"ContainerStarted","Data":"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9"} Oct 30 00:57:36.503029 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.502931 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" event={"ID":"0ccd4ec3-abd3-48fb-a7bf-313016e396a7","Type":"ContainerStarted","Data":"eb978d07dc7f8e95543d07661f0af8a22f63fd2deabb57167ec2cc609dc282b9"} Oct 30 00:57:36.583885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.583786 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podStartSLOduration=1.583767116 podStartE2EDuration="1.583767116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:36.543820329 +0000 UTC m=+847.386380922" watchObservedRunningTime="2026-10-30 00:57:36.583767116 +0000 UTC m=+847.426327710" Oct 30 00:57:36.587912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.585338 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podStartSLOduration=1.585320695 podStartE2EDuration="1.585320695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:36.583403243 +0000 UTC m=+847.425963837" watchObservedRunningTime="2026-10-30 00:57:36.585320695 +0000 UTC m=+847.427881289" Oct 30 00:57:36.685046 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.684960 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:36.903947 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:36.884947 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:37.709884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:37.709005 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:37.709884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:37.709073 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:37.890901 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:37.890581 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:57:37.890901 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:37.890644 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 00:57:38.684884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:38.684245 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:38.684884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:38.684304 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:38.886884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:38.886296 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:57:38.886884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:38.886359 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 00:57:39.700033 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:39.699999 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:39.700325 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:39.700295 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:39.912027 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:39.911989 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:57:39.912636 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:39.912603 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 00:57:40.689885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:40.687257 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:40.689885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:40.687336 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:40.888886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:40.888272 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:57:40.888886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:40.888346 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 00:57:41.688885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:41.686963 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:41.688885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:41.687026 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:41.884833 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:41.884796 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 00:57:41.885203 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:41.885173 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 00:57:42.682571 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:42.682534 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 00:57:42.682745 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:42.682592 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 00:57:43.036973 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.036853 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:43.037542 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.037518 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:43.039095 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.039075 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 00:57:43.722336 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.722307 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:43.722662 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.722643 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:43.723342 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:43.723323 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 00:57:50.785557 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.785522 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 00:57:50.789415 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.789394 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.816464 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.816436 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82b647-openshift-default-dockercfg-w4zsm\"" Oct 30 00:57:50.819482 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.819455 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 00:57:50.833470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833354 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833403 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wb5d\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833428 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833476 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833620 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833642 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833663 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833771 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.833982 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.833944 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.934880 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.934831 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.934896 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wb5d\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.934924 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.934970 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935227 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935045 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935227 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935144 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935227 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935184 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935227 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935222 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935386 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935316 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935386 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935336 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935497 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935477 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935562 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935524 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935783 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935761 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.935896 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.935791 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.937641 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.937619 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.937975 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.937954 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.951524 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.951499 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:50.951823 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:50.951803 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wb5d\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:51.103089 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.103009 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:51.299138 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.299106 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 00:57:51.303558 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.303532 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.324176 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.324151 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 00:57:51.340240 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340210 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340384 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340255 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340384 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340358 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340527 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340401 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340527 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340447 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6dq\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340640 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340575 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340640 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340615 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340751 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340651 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.340810 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.340785 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.360153 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.360071 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fd3281-openshift-default-dockercfg-7q6pm\"" Oct 30 00:57:51.416565 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.416540 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 00:57:51.418151 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:57:51.418128 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e0db78_9933_424d_8460_cd8655b21303.slice/crio-078f39aca87aef53d8d79384b55aa0fd1aa9831934be8e34594b086277758b1f WatchSource:0}: Error finding container 078f39aca87aef53d8d79384b55aa0fd1aa9831934be8e34594b086277758b1f: Status 404 returned error can't find the container with id 078f39aca87aef53d8d79384b55aa0fd1aa9831934be8e34594b086277758b1f Oct 30 00:57:51.420665 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.420631 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:51.420767 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.420696 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:51.420767 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.420736 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:51.441466 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441444 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441578 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441491 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441578 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441522 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441660 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441614 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441660 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441642 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441669 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441695 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441738 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6dq\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.441877 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.441789 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.442155 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.442130 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.442245 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.442156 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.442331 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.442242 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.442403 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.442379 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.442660 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.442637 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.444851 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.444824 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.445055 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.445036 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.459098 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.459071 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.459098 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.459086 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6dq\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.599951 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.599910 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" event={"ID":"81e0db78-9933-424d-8460-cd8655b21303","Type":"ContainerStarted","Data":"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543"} Oct 30 00:57:51.600176 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.599980 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" event={"ID":"81e0db78-9933-424d-8460-cd8655b21303","Type":"ContainerStarted","Data":"078f39aca87aef53d8d79384b55aa0fd1aa9831934be8e34594b086277758b1f"} Oct 30 00:57:51.617565 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.617479 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:51.639112 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.639056 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podStartSLOduration=1.6390411459999998 podStartE2EDuration="1.639041146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:51.635367257 +0000 UTC m=+862.477927851" watchObservedRunningTime="2026-10-30 00:57:51.639041146 +0000 UTC m=+862.481601739" Oct 30 00:57:51.956549 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.956503 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 00:57:51.964158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.964122 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:51.964271 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.964209 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:51.964271 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:51.964262 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:57:52.103652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.103510 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:52.104912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.104848 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:52.105071 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.104929 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:52.625903 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.621723 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" event={"ID":"500a5bc0-29ab-4568-a204-370d70d94966","Type":"ContainerStarted","Data":"abe6b41637f8d963720ee5236fc18776d522bf3ba394b7d26079cb745e4b21f4"} Oct 30 00:57:52.625903 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.621795 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" event={"ID":"500a5bc0-29ab-4568-a204-370d70d94966","Type":"ContainerStarted","Data":"1b5361151bae01b5740e361485a80476a373697e249c218180c5e939a1ff0535"} Oct 30 00:57:52.691884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:52.691141 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podStartSLOduration=1.691122961 podStartE2EDuration="1.691122961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:52.676502593 +0000 UTC m=+863.519063187" watchObservedRunningTime="2026-10-30 00:57:52.691122961 +0000 UTC m=+863.533683579" Oct 30 00:57:53.104885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:53.103967 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:53.104885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:53.104028 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:53.618888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:53.618211 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:53.624918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:53.624001 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:53.624918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:53.624063 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:54.118650 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:54.118212 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:54.118650 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:54.118274 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:54.620237 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:54.620207 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:54.620988 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:54.620957 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:55.112753 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:55.112715 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:55.113071 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:55.113041 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:55.444218 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:57:55.444185 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:57:55.618455 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:55.618413 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:55.618620 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:55.618489 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:56.111886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:56.109967 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:56.111886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:56.110030 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:56.618415 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:56.618372 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:56.618962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:56.618440 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:57.104201 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:57.104167 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 00:57:57.104473 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:57.104445 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 00:57:57.618414 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:57.618372 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:57.618608 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:57.618438 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:58.367428 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:58.367398 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:58.368189 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:58.368168 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:58.369952 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:58.369929 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 00:57:58.617999 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:58.617917 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 00:57:58.617999 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:58.617968 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 00:57:59.623442 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:59.623415 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:59.623925 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:59.623772 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:57:59.624551 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:57:59.624532 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 00:58:21.233351 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.233312 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 00:58:21.237975 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.237955 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.265973 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.265945 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ba3d7b-openshift-default-dockercfg-6zwjk\"" Oct 30 00:58:21.269567 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.269504 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 00:58:21.369467 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369435 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.369670 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369492 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkxj\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.369670 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369639 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.369776 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369720 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.369832 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369807 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.369924 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.369902 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.370019 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.370004 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.370084 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.370046 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.370084 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.370072 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.470938 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.470896 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkxj\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471132 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.470948 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471132 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471003 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471132 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471031 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471132 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471099 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471154 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471193 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471217 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471289 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471624 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471466 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471624 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471583 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.471765 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.471735 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.472051 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.472023 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.472051 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.472039 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.474376 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.474352 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.474996 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.474969 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.489165 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.489078 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkxj\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.489556 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.489530 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.551351 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.551318 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:21.880499 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:58:21.880436 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7989d88_bade_4470_96c5_9c2e07342c5a.slice/crio-8a6c5fa5d186f9bc164533c1be5c9917c28b627074fc194b4300b05c55b7cc3f WatchSource:0}: Error finding container 8a6c5fa5d186f9bc164533c1be5c9917c28b627074fc194b4300b05c55b7cc3f: Status 404 returned error can't find the container with id 8a6c5fa5d186f9bc164533c1be5c9917c28b627074fc194b4300b05c55b7cc3f Oct 30 00:58:21.882907 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.882880 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 00:58:21.884618 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.884578 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:21.884724 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.884664 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:21.884724 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:21.884714 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:22.863792 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:22.863752 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" event={"ID":"f7989d88-bade-4470-96c5-9c2e07342c5a","Type":"ContainerStarted","Data":"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31"} Oct 30 00:58:22.864426 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:22.864400 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" event={"ID":"f7989d88-bade-4470-96c5-9c2e07342c5a","Type":"ContainerStarted","Data":"8a6c5fa5d186f9bc164533c1be5c9917c28b627074fc194b4300b05c55b7cc3f"} Oct 30 00:58:22.940887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:22.940194 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podStartSLOduration=1.9401720089999999 podStartE2EDuration="1.940172009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:22.928357557 +0000 UTC m=+893.770918153" watchObservedRunningTime="2026-10-30 00:58:22.940172009 +0000 UTC m=+893.782732620" Oct 30 00:58:23.552359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:23.552316 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:23.553851 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:23.553830 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:23.553941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:23.553890 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:24.555900 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:24.552339 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:24.555900 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:24.552399 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:25.426902 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:58:25.426832 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:58:25.552577 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:25.552540 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:25.552779 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:25.552604 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:26.551996 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:26.551966 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:26.552415 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:26.552017 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:27.551977 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:27.551942 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:27.552161 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:27.551993 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:28.552309 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:28.552274 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 00:58:28.552711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:28.552326 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 00:58:29.389909 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:29.389878 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:58:29.394806 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:29.394778 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 00:58:29.556976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:29.556950 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:29.557432 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:29.557323 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:29.558126 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:29.558110 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 00:58:49.893909 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:49.893796 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 00:58:49.898394 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:49.898370 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:49.923489 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:49.923463 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d827b7\"/\"distinct-2-w4-d827b7-kserve-self-signed-certs\"" Oct 30 00:58:49.926011 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:49.925987 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 00:58:50.063047 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063007 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.063245 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063124 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.063245 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063177 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjc2j\" (UniqueName: \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.063349 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063324 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.063394 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063351 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.063455 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.063395 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165016 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.164913 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165180 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165027 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165180 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165137 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165191 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjc2j\" (UniqueName: \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165240 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165271 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165516 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165486 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165599 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165548 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.165668 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.165650 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.167402 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.167375 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.168437 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.168413 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.182705 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.182679 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjc2j\" (UniqueName: \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\") pod \"distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.213737 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.213707 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:50.513805 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:50.513778 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 00:58:50.516327 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:58:50.516295 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d360b3e_9afb_4f29_abd4_eef10ce7a89c.slice/crio-0a6cb1be2927cc5fd4d3432809dacdcdbae4b7d466b446df30df2f9abb28f73a WatchSource:0}: Error finding container 0a6cb1be2927cc5fd4d3432809dacdcdbae4b7d466b446df30df2f9abb28f73a: Status 404 returned error can't find the container with id 0a6cb1be2927cc5fd4d3432809dacdcdbae4b7d466b446df30df2f9abb28f73a Oct 30 00:58:51.051929 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.051890 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" event={"ID":"9d360b3e-9afb-4f29-abd4-eef10ce7a89c","Type":"ContainerStarted","Data":"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d"} Oct 30 00:58:51.051929 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.051933 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" event={"ID":"9d360b3e-9afb-4f29-abd4-eef10ce7a89c","Type":"ContainerStarted","Data":"0a6cb1be2927cc5fd4d3432809dacdcdbae4b7d466b446df30df2f9abb28f73a"} Oct 30 00:58:51.060985 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.060952 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 00:58:51.064701 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.064678 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.079607 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.079587 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d827b7-openshift-default-dockercfg-95vfg\"" Oct 30 00:58:51.091118 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.091098 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 00:58:51.092018 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.091974 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podStartSLOduration=2.091958825 podStartE2EDuration="2.091958825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:51.090099931 +0000 UTC m=+921.932660525" watchObservedRunningTime="2026-10-30 00:58:51.091958825 +0000 UTC m=+921.934519436" Oct 30 00:58:51.176054 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176022 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176230 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176079 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn48c\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176280 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176253 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176335 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176309 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176463 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176443 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176547 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176519 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176665 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176596 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176735 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176683 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.176976 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.176823 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.209193 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.209166 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 00:58:51.213819 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.213786 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.217577 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.213851 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:58:51.227821 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.227796 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c4847c-openshift-default-dockercfg-4nf8k\"" Oct 30 00:58:51.239224 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.239193 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 00:58:51.277458 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277425 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277485 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277518 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277544 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn48c\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277582 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277644 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277623 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277663 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277728 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277769 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.277961 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.277951 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.278169 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.278025 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.278169 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.278140 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.278258 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.278226 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.278732 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.278697 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.280345 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.280315 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.280345 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.280336 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.297131 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.297104 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.297256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.297189 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn48c\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.377854 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.377777 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:51.379230 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379198 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379266 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379359 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379316 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcx2\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379445 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379417 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379486 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379436 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379486 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379468 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379583 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379701 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379689 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.379735 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.379715 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.487953 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.487922 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.487980 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488013 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcx2\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488049 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488064 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488156 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488098 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488328 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488163 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488328 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488235 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488328 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488270 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488434 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488402 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.488837 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.488811 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.489083 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.489062 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.489158 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.489070 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.489293 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.489273 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.490844 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.490806 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.491714 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.491696 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.504772 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.504748 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.505094 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.505072 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcx2\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.531703 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.531678 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:51.697645 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.697616 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 00:58:51.698473 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:58:51.698447 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0d01bd_dd62_4ac8_a647_bd8214cc15e4.slice/crio-dd484ae6c3a3b49daf579c46f1f9cbeb7633fe774a2b3094ad94895eb4c8f0dd WatchSource:0}: Error finding container dd484ae6c3a3b49daf579c46f1f9cbeb7633fe774a2b3094ad94895eb4c8f0dd: Status 404 returned error can't find the container with id dd484ae6c3a3b49daf579c46f1f9cbeb7633fe774a2b3094ad94895eb4c8f0dd Oct 30 00:58:51.701030 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.700996 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:51.701130 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.701072 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:51.701130 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.701117 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:51.835602 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.835578 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 00:58:51.837671 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:58:51.837638 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e735e85_42d3_4caa_a9be_33445bfa6d0b.slice/crio-117af7768ee4813dbf63a40811f9854a48f981c7c9f0513a1474836fd9911406 WatchSource:0}: Error finding container 117af7768ee4813dbf63a40811f9854a48f981c7c9f0513a1474836fd9911406: Status 404 returned error can't find the container with id 117af7768ee4813dbf63a40811f9854a48f981c7c9f0513a1474836fd9911406 Oct 30 00:58:51.840574 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.840532 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:51.840661 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.840623 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:51.840705 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:51.840671 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:58:52.060971 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.060871 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" event={"ID":"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4","Type":"ContainerStarted","Data":"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121"} Oct 30 00:58:52.060971 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.060941 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" event={"ID":"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4","Type":"ContainerStarted","Data":"dd484ae6c3a3b49daf579c46f1f9cbeb7633fe774a2b3094ad94895eb4c8f0dd"} Oct 30 00:58:52.062981 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.062952 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" event={"ID":"4e735e85-42d3-4caa-a9be-33445bfa6d0b","Type":"ContainerStarted","Data":"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108"} Oct 30 00:58:52.063114 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.062998 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" event={"ID":"4e735e85-42d3-4caa-a9be-33445bfa6d0b","Type":"ContainerStarted","Data":"117af7768ee4813dbf63a40811f9854a48f981c7c9f0513a1474836fd9911406"} Oct 30 00:58:52.098832 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.098591 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podStartSLOduration=1.098571893 podStartE2EDuration="1.098571893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:52.096941357 +0000 UTC m=+922.939501969" watchObservedRunningTime="2026-10-30 00:58:52.098571893 +0000 UTC m=+922.941132485" Oct 30 00:58:52.134092 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.134028 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podStartSLOduration=1.134012245 podStartE2EDuration="1.134012245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:52.13313481 +0000 UTC m=+922.975695403" watchObservedRunningTime="2026-10-30 00:58:52.134012245 +0000 UTC m=+922.976572838" Oct 30 00:58:52.378120 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.378031 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:52.379604 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.379576 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:52.379721 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.379641 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:52.535164 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.533205 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:52.538274 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.536883 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:52.538274 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:52.536952 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:53.396885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:53.395147 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:53.396885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:53.395226 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:53.532761 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:53.532719 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:53.532990 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:53.532777 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:54.381885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:54.381220 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:54.381885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:54.381285 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:54.532253 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:54.532087 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:54.532253 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:54.532157 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:55.379289 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:55.379247 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:55.379600 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:55.379569 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:55.511974 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:58:55.511938 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:58:55.532201 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:55.532174 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:55.532429 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:55.532408 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:56.378241 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:56.378206 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:56.378522 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:56.378494 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:56.532601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:56.532570 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:56.533018 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:56.532622 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:57.388627 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:57.388590 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:57.388958 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:57.388926 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:57.532630 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:57.532598 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:57.533044 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:57.532948 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:58.378571 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:58.378529 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 00:58:58.378760 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:58.378596 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 00:58:58.532511 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:58.532479 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 00:58:58.532695 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:58.532539 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 00:58:59.383290 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.383255 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:59.383666 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.383639 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:59.384547 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.384518 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 00:58:59.638463 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.638386 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:59.638939 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.638737 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:58:59.640757 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:58:59.640733 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 00:59:00.214258 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:00.214225 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:59:00.227019 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:00.226997 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:59:01.194705 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:01.194674 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 00:59:25.427724 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:59:25.427683 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:59:36.195909 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.195877 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 00:59:36.200438 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.200417 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.224073 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.224045 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 00:59:36.281647 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281615 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.281797 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281663 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.281797 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281775 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nvl\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.281906 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281797 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.281906 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281878 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.281986 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281943 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.282020 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.281999 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.282054 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.282043 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.282181 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.282161 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.331197 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.331155 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 00:59:36.335387 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.335363 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.358393 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.358364 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 00:59:36.363749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.363719 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 00:59:36.368514 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.368488 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.388084 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.388055 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 00:59:36.389291 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389265 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.389436 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389354 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nvl\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389543 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389600 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389649 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389732 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389795 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389874 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389917 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.389945 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.390678 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.391107 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.395249 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.401730 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.414312 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.411107 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.446928 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.444309 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.471883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.469242 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491081 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491137 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491171 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491229 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491347 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491404 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491456 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wd4\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491493 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491546 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491570 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491598 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491669 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491719 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491752 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491776 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491805 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491839 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.491930 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.491901 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zt4\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.500328 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.494936 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nvl\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.518703 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.518663 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.592874 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.592942 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wd4\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.592978 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593027 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593052 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593078 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593138 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593183 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593217 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593240 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593265 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593297 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593313 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593339 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zt4\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593408 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593461 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593507 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.593615 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593564 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.594877 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.593986 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.594877 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.594233 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.594877 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.594802 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.595057 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.594885 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.595120 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.595078 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.601918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.595330 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.601918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.595541 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.601918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.596108 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.601918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.596289 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.601918 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.600471 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.603608 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.603585 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.606285 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.606265 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.606447 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.606426 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.610658 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.610633 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.611135 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.611112 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.611308 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.611291 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wd4\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.611774 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.611758 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zt4\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.652195 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.652152 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:36.695314 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.695285 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:36.846326 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:59:36.846296 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c42d62_0275_417c_932e_de81098d49df.slice/crio-0a7106f549eed91ee0437514e760c280546bc1efd2483c09c8b60bd5420aa2f6 WatchSource:0}: Error finding container 0a7106f549eed91ee0437514e760c280546bc1efd2483c09c8b60bd5420aa2f6: Status 404 returned error can't find the container with id 0a7106f549eed91ee0437514e760c280546bc1efd2483c09c8b60bd5420aa2f6 Oct 30 00:59:36.849204 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.849177 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 00:59:36.849547 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.849510 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:36.849640 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.849582 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:36.849640 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.849618 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:36.962413 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.962383 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 00:59:36.965248 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:59:36.965202 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d569248_d28e_4f24_8b97_dad688dde3ca.slice/crio-0f14b4b25640fc5734d81e613a120be9130c851e1ef1cbedbc39bd15fbb7961f WatchSource:0}: Error finding container 0f14b4b25640fc5734d81e613a120be9130c851e1ef1cbedbc39bd15fbb7961f: Status 404 returned error can't find the container with id 0f14b4b25640fc5734d81e613a120be9130c851e1ef1cbedbc39bd15fbb7961f Oct 30 00:59:36.969400 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.969367 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:36.969499 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.969439 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:36.969499 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:36.969469 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:37.009391 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.009365 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 00:59:37.012034 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:59:37.012000 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15da3c6a_3e23_4a89_b1a8_29b6f4c919f8.slice/crio-45b75a349bdced25be322e74d20a46434db6c4592cc648f018cfb09b75bb4205 WatchSource:0}: Error finding container 45b75a349bdced25be322e74d20a46434db6c4592cc648f018cfb09b75bb4205: Status 404 returned error can't find the container with id 45b75a349bdced25be322e74d20a46434db6c4592cc648f018cfb09b75bb4205 Oct 30 00:59:37.014933 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.014898 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:37.015064 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.014977 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:37.015064 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.015015 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:37.498542 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.498496 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" event={"ID":"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8","Type":"ContainerStarted","Data":"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77"} Oct 30 00:59:37.499054 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.498577 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" event={"ID":"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8","Type":"ContainerStarted","Data":"45b75a349bdced25be322e74d20a46434db6c4592cc648f018cfb09b75bb4205"} Oct 30 00:59:37.500551 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.500516 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" event={"ID":"d9c42d62-0275-417c-932e-de81098d49df","Type":"ContainerStarted","Data":"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162"} Oct 30 00:59:37.500768 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.500563 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" event={"ID":"d9c42d62-0275-417c-932e-de81098d49df","Type":"ContainerStarted","Data":"0a7106f549eed91ee0437514e760c280546bc1efd2483c09c8b60bd5420aa2f6"} Oct 30 00:59:37.502329 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.502309 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" event={"ID":"9d569248-d28e-4f24-8b97-dad688dde3ca","Type":"ContainerStarted","Data":"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2"} Oct 30 00:59:37.502436 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.502339 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" event={"ID":"9d569248-d28e-4f24-8b97-dad688dde3ca","Type":"ContainerStarted","Data":"0f14b4b25640fc5734d81e613a120be9130c851e1ef1cbedbc39bd15fbb7961f"} Oct 30 00:59:37.519042 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.519003 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:37.520511 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.520488 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:37.520646 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.520538 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:37.600885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.599971 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podStartSLOduration=1.5999520889999999 podStartE2EDuration="1.599952089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.552486032 +0000 UTC m=+968.395046712" watchObservedRunningTime="2026-10-30 00:59:37.599952089 +0000 UTC m=+968.442512684" Oct 30 00:59:37.621887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.621104 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podStartSLOduration=1.621076623 podStartE2EDuration="1.621076623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.597391675 +0000 UTC m=+968.439952279" watchObservedRunningTime="2026-10-30 00:59:37.621076623 +0000 UTC m=+968.463637248" Oct 30 00:59:37.653887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.653069 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:37.656883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.655352 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podStartSLOduration=1.655318459 podStartE2EDuration="1.655318459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.637220159 +0000 UTC m=+968.479780753" watchObservedRunningTime="2026-10-30 00:59:37.655318459 +0000 UTC m=+968.497879054" Oct 30 00:59:37.696983 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:37.696948 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:38.533145 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.533110 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:38.542218 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.542100 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:38.654459 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.654427 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:38.654654 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.654635 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:38.703886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.703774 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:38.704087 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:38.703900 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:39.535888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.533611 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:39.535888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.533675 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:39.656885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.656807 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:39.657090 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.656906 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:39.722887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.705296 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:39.722887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:39.722171 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:40.519888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.519287 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:40.519888 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.519352 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:40.653887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.653291 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:40.653887 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.653363 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:40.696886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.696610 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:40.696886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:40.696672 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:41.221011 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.220966 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:41.521351 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.521259 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:41.521587 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.521558 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:41.652697 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.652667 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:41.652952 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.652927 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:41.697374 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.697342 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:41.697951 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:41.697923 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:42.519300 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.519260 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:42.519632 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.519602 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:42.658529 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.658496 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:42.658828 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.658783 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:42.732886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.728462 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:42.732886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:42.728523 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:43.519845 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.519800 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 00:59:43.519845 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.519873 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 00:59:43.653288 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.653254 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 00:59:43.653481 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.653307 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 00:59:43.696634 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.696590 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 00:59:43.696803 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:43.696655 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 00:59:44.657194 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.657166 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:44.657675 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.657467 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:44.658470 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.658450 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 00:59:44.714783 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.714752 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:44.715216 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.715197 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:44.716939 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:44.716920 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 00:59:45.520061 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:45.520014 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:59:45.520270 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:45.520104 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:46.292427 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:46.292395 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:46.292912 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:46.292813 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:46.293915 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:46.293891 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 00:59:51.250541 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.250503 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 00:59:51.255463 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.255437 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.292306 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.292266 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 00:59:51.374465 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374431 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374472 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374619 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374534 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hgf\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374739 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374663 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374739 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374707 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374739 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374735 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.374962 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374944 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.375029 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.374969 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.375029 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.375006 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476062 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476028 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476084 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476116 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476148 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476169 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476203 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476239 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476229 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476515 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476267 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476515 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476321 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hgf\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476711 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476684 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476796 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476717 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476796 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476738 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.476936 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.476810 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.477350 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.477327 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.478706 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.478682 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.478877 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.478843 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.493874 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.493824 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.494119 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.494100 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hgf\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.596398 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.596326 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:51.680749 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.680717 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 00:59:51.685458 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.685436 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.733824 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.733789 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 00:59:51.779834 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.779808 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780076 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.779910 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjtp\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780076 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.779966 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780076 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780062 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780105 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780124 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780149 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780201 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.780256 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.780222 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.786795 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.786755 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 00:59:51.791652 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.791634 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.813656 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.813605 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 00:59:51.881214 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881111 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881214 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881185 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.881467 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881236 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjtp\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881467 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881290 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881467 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881421 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.881629 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881512 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881629 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881528 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmdk\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.881741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881639 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881666 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.881741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881713 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881741 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881713 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881949 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881746 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.881949 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881787 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.881949 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881814 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.882105 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881955 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882105 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.881989 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.882105 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882018 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.882105 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882031 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882105 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882087 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882357 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882088 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882357 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882195 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882460 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882441 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.882524 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.882475 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.884501 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.884477 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.884638 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.884619 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.897954 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.897926 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjtp\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.898219 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.898199 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:51.914900 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.912680 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 00:59:51.922937 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.922891 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:51.923120 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.922982 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:51.923120 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.923032 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984296 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984404 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmdk\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984459 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984511 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984539 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984791 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984847 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984940 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984954 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.985069 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.985327 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.984736 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.986160 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.985376 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.986160 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.985636 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.986160 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.985801 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.989249 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.989204 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:51.990404 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:51.990351 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:52.004001 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.003755 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:52.004936 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.004843 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmdk\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:52.006601 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.006564 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:52.155457 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.154296 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:52.430911 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.428544 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 00:59:52.466125 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.465457 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.466125 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.465573 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.466125 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.465659 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.539984 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.539950 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 00:59:52.555622 ip-10-0-143-7 kubenswrapper[5548]: W1030 00:59:52.555579 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86979f2_c453_41b0_aa44_efbae0f70fb0.slice/crio-f1f6a759b4a17d9ff2b6b659c610e0ede198d0fadf0e1fcd39af401101b5935e WatchSource:0}: Error finding container f1f6a759b4a17d9ff2b6b659c610e0ede198d0fadf0e1fcd39af401101b5935e: Status 404 returned error can't find the container with id f1f6a759b4a17d9ff2b6b659c610e0ede198d0fadf0e1fcd39af401101b5935e Oct 30 00:59:52.560086 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.559308 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.560086 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.559406 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.560086 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.559452 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 00:59:52.739883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.729579 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" event={"ID":"e0707bbd-1bd8-4708-89a1-f2f350fe50f3","Type":"ContainerStarted","Data":"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79"} Oct 30 00:59:52.739883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.729645 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" event={"ID":"e0707bbd-1bd8-4708-89a1-f2f350fe50f3","Type":"ContainerStarted","Data":"5c58b685757c5f33cca0213a99f8d64d55ca348c65e38f99948c284baf77437f"} Oct 30 00:59:52.754045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.750377 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" event={"ID":"31fc7b71-ddec-475b-921c-4c4708bc7ef7","Type":"ContainerStarted","Data":"f6855283695e146196fca24d65a59f459208408991c5ffdb246f44b4cd93d2a3"} Oct 30 00:59:52.754045 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.750444 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" event={"ID":"31fc7b71-ddec-475b-921c-4c4708bc7ef7","Type":"ContainerStarted","Data":"e35aa030be75d2c5ba6fd6a1a2de0be0b3f127a4e9a02c8e25e17fa9951842ce"} Oct 30 00:59:52.771900 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.768334 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" event={"ID":"f86979f2-c453-41b0-aa44-efbae0f70fb0","Type":"ContainerStarted","Data":"f1f6a759b4a17d9ff2b6b659c610e0ede198d0fadf0e1fcd39af401101b5935e"} Oct 30 00:59:52.809910 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:52.807431 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podStartSLOduration=1.8074127739999999 podStartE2EDuration="1.807412774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.802483273 +0000 UTC m=+983.645043867" watchObservedRunningTime="2026-10-30 00:59:52.807412774 +0000 UTC m=+983.649973370" Oct 30 00:59:53.012941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.010005 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 00:59:53.012941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.011700 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:53.012941 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.011956 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:53.611412 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.597882 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 00:59:53.618816 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.612240 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:53.618816 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.612322 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:53.833883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.826007 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" event={"ID":"f86979f2-c453-41b0-aa44-efbae0f70fb0","Type":"ContainerStarted","Data":"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92"} Oct 30 00:59:53.875948 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:53.873829 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podStartSLOduration=2.873809596 podStartE2EDuration="2.873809596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.855152539 +0000 UTC m=+983.697713130" watchObservedRunningTime="2026-10-30 00:59:53.873809596 +0000 UTC m=+984.716370190" Oct 30 00:59:54.008883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.008305 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:54.008883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.008370 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:54.115884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.106952 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.164895 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.164404 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 00:59:54.164895 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.164549 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:54.164895 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.164593 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:54.451939 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.451755 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 00:59:54.601301 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.601264 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:54.602736 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.602703 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:54.728040 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:54.727943 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:55.032111 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.031419 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:55.032111 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.031486 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:55.109048 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.108999 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:55.110030 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.110007 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:59:55.150356 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.150283 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podStartSLOduration=4.150260349 podStartE2EDuration="4.150260349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:53.880179939 +0000 UTC m=+984.722740533" watchObservedRunningTime="2026-10-30 00:59:55.150260349 +0000 UTC m=+985.992820964" Oct 30 00:59:55.291528 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.291438 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:55.291717 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.291599 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:55.294920 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.292826 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:59:55.324884 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.304653 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 00:59:55.325643 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.309033 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 00:59:55.326107 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.311674 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 00:59:55.474886 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.458437 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:55.586035 ip-10-0-143-7 kubenswrapper[5548]: E1030 00:59:55.585914 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 00:59:55.612927 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.599290 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:55.612927 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.599348 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:55.618316 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:55.618295 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:59:56.043044 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.038217 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:56.043044 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.038778 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:56.043044 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.038824 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:56.115092 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.115039 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:56.175075 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.174950 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:56.175426 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.175385 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:56.345937 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.345421 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:56.345937 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.345741 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:56.346297 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.346026 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 00:59:56.349891 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.346514 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:59:56.349891 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.346571 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:59:56.349891 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.346609 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:59:56.491433 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.491392 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 00:59:56.632747 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.632667 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:56.633053 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.633025 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:56.689290 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.689263 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 00:59:56.786321 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.786289 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 00:59:56.864945 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:56.864914 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 00:59:57.088815 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.088781 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:57.089518 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.089487 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:57.163664 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.163619 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:57.163883 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.163690 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:57.164983 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.164959 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 00:59:57.188885 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.182355 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 00:59:57.600693 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.600657 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:57.601036 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:57.601003 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:58.010081 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.010046 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:58.010388 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.010360 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:58.162844 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.162604 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:58.162844 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.162666 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:58.609446 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.609400 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:58.609668 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:58.609489 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 00:59:59.013285 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.013244 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:59:59.013584 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.013553 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 00:59:59.162466 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.162427 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:59:59.162630 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.162478 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 00:59:59.629609 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.629573 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 00:59:59.649491 ip-10-0-143-7 kubenswrapper[5548]: I1030 00:59:59.649450 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:00:00.005325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.005293 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:00:00.005612 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.005587 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:00:00.149701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.149670 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:00.154179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.154156 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:00.154307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.154281 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.158005 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.157919 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.163053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.162608 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 01:00:00.163053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.162673 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 01:00:00.194400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.194368 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:00:00.202886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.201951 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:00.222055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.221952 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.222470 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.222423 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsfc\" (UniqueName: \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.222745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.222728 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qp7\" (UniqueName: \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.232915 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.230001 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.345819 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.345922 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.346013 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsfc\" (UniqueName: \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.346071 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qp7\" (UniqueName: \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.347012 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.347622 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.349065 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.349123 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.349154 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:00.364060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.349252 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.443410 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.443375 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsfc\" (UniqueName: \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.453984 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.453953 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qp7\" (UniqueName: \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.455276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.455176 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvfb\" (UniqueName: \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.455704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.455676 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.502905 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.502843 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:00.557273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.557228 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvfb\" (UniqueName: \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.557500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.557468 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.558321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.558297 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.606977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.606491 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:00:00.606977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.606568 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 01:00:00.610677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.610646 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvfb\" (UniqueName: \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\") pod \"maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.622761 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.622500 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:00.755431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.755401 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:00.894985 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:00:00.894938 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ff02b9_9a9c_48de_9288_9da9d9e5548b.slice/crio-202ffa6d5a3021e330564122b259865377caa25723f47f8803220d172933d3a1 WatchSource:0}: Error finding container 202ffa6d5a3021e330564122b259865377caa25723f47f8803220d172933d3a1: Status 404 returned error can't find the container with id 202ffa6d5a3021e330564122b259865377caa25723f47f8803220d172933d3a1 Oct 30 01:00:00.898200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.898078 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:00.985787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:00.985737 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:00.990686 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:00:00.990644 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cae984_2e6a_44af_b0a9_bf889c20cf15.slice/crio-f5f1add05bb50773b61306e8b5b3ba3664c2b00b6ad5e276eb39d965e13024f6 WatchSource:0}: Error finding container f5f1add05bb50773b61306e8b5b3ba3664c2b00b6ad5e276eb39d965e13024f6: Status 404 returned error can't find the container with id f5f1add05bb50773b61306e8b5b3ba3664c2b00b6ad5e276eb39d965e13024f6 Oct 30 01:00:01.005670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.005643 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:00:01.005802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.005711 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:00:01.095751 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.095299 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:01.097905 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:00:01.097873 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f91aac_4ed2_423a_bbbf_721daee73586.slice/crio-e695906a8bd866c762b12db80f1e2750d071d54bb49b3d98168fd179faf4be1d WatchSource:0}: Error finding container e695906a8bd866c762b12db80f1e2750d071d54bb49b3d98168fd179faf4be1d: Status 404 returned error can't find the container with id e695906a8bd866c762b12db80f1e2750d071d54bb49b3d98168fd179faf4be1d Oct 30 01:00:01.163248 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.163214 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 01:00:01.163441 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.163274 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 01:00:01.429592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.429493 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerStarted","Data":"f5f1add05bb50773b61306e8b5b3ba3664c2b00b6ad5e276eb39d965e13024f6"} Oct 30 01:00:01.432200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.432137 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" event={"ID":"98ff02b9-9a9c-48de-9288-9da9d9e5548b","Type":"ContainerStarted","Data":"202ffa6d5a3021e330564122b259865377caa25723f47f8803220d172933d3a1"} Oct 30 01:00:01.434421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:01.434380 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerStarted","Data":"e695906a8bd866c762b12db80f1e2750d071d54bb49b3d98168fd179faf4be1d"} Oct 30 01:00:02.120886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.120825 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 01:00:02.122640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.122576 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 01:00:02.124661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.124645 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 01:00:02.417587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.417556 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 01:00:02.418162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.418135 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 01:00:02.420360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.420317 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 01:00:02.478540 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.478508 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 01:00:02.479901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.479842 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 01:00:02.481016 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:02.480995 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 01:00:05.475303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.475268 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerStarted","Data":"e66adecb268027f835619e2d539bcfd753ca61d6ca6848ca6808c03fa2ba8e5b"} Oct 30 01:00:05.477088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.477062 5548 generic.go:356] "Generic (PLEG): container finished" podID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerID="cb34475d1b89a55bb4cbcd87aa97cae2ca46eb8ba35152d72785a29c7e2b05eb" exitCode=60 Oct 30 01:00:05.477231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.477144 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" event={"ID":"98ff02b9-9a9c-48de-9288-9da9d9e5548b","Type":"ContainerDied","Data":"cb34475d1b89a55bb4cbcd87aa97cae2ca46eb8ba35152d72785a29c7e2b05eb"} Oct 30 01:00:05.477402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.477383 5548 scope.go:122] "RemoveContainer" containerID="cb34475d1b89a55bb4cbcd87aa97cae2ca46eb8ba35152d72785a29c7e2b05eb" Oct 30 01:00:05.479069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.479003 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerStarted","Data":"103b135487e07e47bbba99bd04d73a19e971bd23a8b42e861811c86b173df2c5"} Oct 30 01:00:05.479251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.479235 5548 scope.go:122] "RemoveContainer" containerID="103b135487e07e47bbba99bd04d73a19e971bd23a8b42e861811c86b173df2c5" Oct 30 01:00:05.508274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.508227 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" podStartSLOduration=1.735738564 podStartE2EDuration="5.508212965s" totalImagesPullingTime="3.772474401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:05.506724737 +0000 UTC m=+996.349285348" watchObservedRunningTime="2026-10-30 01:00:05.508212965 +0000 UTC m=+996.350773557" Oct 30 01:00:05.976928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.976901 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:00:05.980893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:05.980842 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.008543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.008516 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:00:06.143516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143480 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.143683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143544 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.143683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143663 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.143829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143708 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.143829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143728 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.143829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143828 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.144059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.143883 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.144059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.144038 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfkx\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.144172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.144079 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245245 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245288 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245332 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245360 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245378 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245509 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245564 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245661 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfkx\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245689 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245698 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245727 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.245818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.245782 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.246105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.246026 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.246355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.246330 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.247979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.247959 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.248183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.248165 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.266280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.266259 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfkx\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.266376 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.266289 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.300424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.300398 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:06.488880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.488829 5548 generic.go:356] "Generic (PLEG): container finished" podID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerID="e66adecb268027f835619e2d539bcfd753ca61d6ca6848ca6808c03fa2ba8e5b" exitCode=60 Oct 30 01:00:06.489312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.488899 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerDied","Data":"e66adecb268027f835619e2d539bcfd753ca61d6ca6848ca6808c03fa2ba8e5b"} Oct 30 01:00:06.489312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.489223 5548 scope.go:122] "RemoveContainer" containerID="e66adecb268027f835619e2d539bcfd753ca61d6ca6848ca6808c03fa2ba8e5b" Oct 30 01:00:06.491267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.491034 5548 generic.go:356] "Generic (PLEG): container finished" podID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerID="f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2" exitCode=60 Oct 30 01:00:06.491267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.491123 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" event={"ID":"98ff02b9-9a9c-48de-9288-9da9d9e5548b","Type":"ContainerDied","Data":"f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2"} Oct 30 01:00:06.491267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.491160 5548 scope.go:122] "RemoveContainer" containerID="cb34475d1b89a55bb4cbcd87aa97cae2ca46eb8ba35152d72785a29c7e2b05eb" Oct 30 01:00:06.491556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.491540 5548 scope.go:122] "RemoveContainer" containerID="f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2" Oct 30 01:00:06.491954 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:06.491815 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v_odh-ai-gateway-infra(98ff02b9-9a9c-48de-9288-9da9d9e5548b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" Oct 30 01:00:06.494043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.494018 5548 generic.go:356] "Generic (PLEG): container finished" podID="26f91aac-4ed2-423a-bbbf-721daee73586" containerID="103b135487e07e47bbba99bd04d73a19e971bd23a8b42e861811c86b173df2c5" exitCode=60 Oct 30 01:00:06.494135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.494096 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerDied","Data":"103b135487e07e47bbba99bd04d73a19e971bd23a8b42e861811c86b173df2c5"} Oct 30 01:00:06.494207 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.494154 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerStarted","Data":"d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9"} Oct 30 01:00:06.556338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.556296 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" podStartSLOduration=3.184198362 podStartE2EDuration="6.556268976s" totalImagesPullingTime="3.372070614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:06.553742205 +0000 UTC m=+997.396302798" watchObservedRunningTime="2026-10-30 01:00:06.556268976 +0000 UTC m=+997.398829593" Oct 30 01:00:06.635870 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.635818 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:00:06.670460 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:00:06.670427 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e6d65d_6cf9_454e_bb91_342dc4158dfb.slice/crio-546a231c81ae7fd5fc03f1a838adce62c2220d88a44fc45a544a727b928e0335 WatchSource:0}: Error finding container 546a231c81ae7fd5fc03f1a838adce62c2220d88a44fc45a544a727b928e0335: Status 404 returned error can't find the container with id 546a231c81ae7fd5fc03f1a838adce62c2220d88a44fc45a544a727b928e0335 Oct 30 01:00:06.673181 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.673151 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:06.673262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.673215 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:06.673262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:06.673244 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:07.505262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.505230 5548 generic.go:356] "Generic (PLEG): container finished" podID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerID="97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89" exitCode=60 Oct 30 01:00:07.505672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.505318 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerDied","Data":"97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89"} Oct 30 01:00:07.505672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.505387 5548 scope.go:122] "RemoveContainer" containerID="e66adecb268027f835619e2d539bcfd753ca61d6ca6848ca6808c03fa2ba8e5b" Oct 30 01:00:07.505758 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.505697 5548 scope.go:122] "RemoveContainer" containerID="97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89" Oct 30 01:00:07.506030 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:07.506011 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh_odh-ai-gateway-infra(c3cae984-2e6a-44af-b0a9-bf889c20cf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" Oct 30 01:00:07.508587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.508570 5548 scope.go:122] "RemoveContainer" containerID="f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2" Oct 30 01:00:07.508905 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:07.508884 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v_odh-ai-gateway-infra(98ff02b9-9a9c-48de-9288-9da9d9e5548b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" Oct 30 01:00:07.510496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.510475 5548 generic.go:356] "Generic (PLEG): container finished" podID="26f91aac-4ed2-423a-bbbf-721daee73586" containerID="d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9" exitCode=60 Oct 30 01:00:07.510604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.510565 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerDied","Data":"d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9"} Oct 30 01:00:07.510772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.510757 5548 scope.go:122] "RemoveContainer" containerID="d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9" Oct 30 01:00:07.511090 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:07.511070 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8_odh-ai-gateway-infra(26f91aac-4ed2-423a-bbbf-721daee73586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" Oct 30 01:00:07.512431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.512407 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" event={"ID":"60e6d65d-6cf9-454e-bb91-342dc4158dfb","Type":"ContainerStarted","Data":"df92edefc8fc824ba21eaeb082ad85a62c81853141aa37f6458b30f5569474ca"} Oct 30 01:00:07.512546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.512448 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" event={"ID":"60e6d65d-6cf9-454e-bb91-342dc4158dfb","Type":"ContainerStarted","Data":"546a231c81ae7fd5fc03f1a838adce62c2220d88a44fc45a544a727b928e0335"} Oct 30 01:00:07.519081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.519065 5548 scope.go:122] "RemoveContainer" containerID="103b135487e07e47bbba99bd04d73a19e971bd23a8b42e861811c86b173df2c5" Oct 30 01:00:07.656682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:07.656628 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podStartSLOduration=2.656612453 podStartE2EDuration="2.656612453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:07.654053933 +0000 UTC m=+998.496614528" watchObservedRunningTime="2026-10-30 01:00:07.656612453 +0000 UTC m=+998.499173047" Oct 30 01:00:08.300971 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:08.300935 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:08.302426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:08.302401 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:08.302481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:08.302457 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:08.521580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:08.521552 5548 scope.go:122] "RemoveContainer" containerID="97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89" Oct 30 01:00:08.522051 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:08.521926 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh_odh-ai-gateway-infra(c3cae984-2e6a-44af-b0a9-bf889c20cf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" Oct 30 01:00:08.523364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:08.523338 5548 scope.go:122] "RemoveContainer" containerID="d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9" Oct 30 01:00:08.523603 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:08.523581 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8_odh-ai-gateway-infra(26f91aac-4ed2-423a-bbbf-721daee73586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" Oct 30 01:00:09.301109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:09.301077 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:09.301287 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:09.301131 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:10.303374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:10.303277 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:10.303374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:10.303356 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:11.306890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:11.306765 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:11.306890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:11.306823 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:12.301887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:12.301273 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:12.301887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:12.301351 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:12.532027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:12.531985 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:12.873009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:12.872960 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.178886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.177962 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.203887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.184201 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:00:13.203887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.184554 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.301884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.301270 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:13.301884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.301346 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:13.502940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.501967 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.779937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.779526 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:00:13.932884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.932336 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.940169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:13.940138 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:00:14.310885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.310805 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:14.311093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.310890 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:14.332920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.332301 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.332920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.332906 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:00:14.357887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.333544 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:00:14.357887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.333653 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:00:14.382436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.382398 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:14.382704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.382677 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.542317 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.522755 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:00:14.542317 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.525368 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:00:14.784462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.783655 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.786562 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.786005 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:00:14.878596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.878521 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 01:00:14.973135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:14.973093 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:00:15.389886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.382024 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.389886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.383247 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:00:15.434266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.434232 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:15.451888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.438985 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:15.484529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.484498 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:15.484760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.484734 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.575979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.575938 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.674392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.674356 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:15.683985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.683941 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.758039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.758000 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:15.758489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.758440 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.761759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.761734 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:00:15.840033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.839991 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.870557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.869425 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:00:15.885899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.870833 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:15.885899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.870901 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.986131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:15.986023 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:16.330601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.330512 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:16.365069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.365034 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 01:00:16.388925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.387550 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:16.389853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.389728 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 01:00:16.504400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.504368 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:00:16.750654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.748424 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:00:16.769903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:16.767249 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:00:17.487723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:17.487688 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:17.488361 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:17.488307 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:17.489805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:17.489786 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:00:19.264914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:19.264887 5548 scope.go:122] "RemoveContainer" containerID="d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9" Oct 30 01:00:20.257407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:20.257381 5548 scope.go:122] "RemoveContainer" containerID="97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89" Oct 30 01:00:20.312030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:20.312006 5548 generic.go:356] "Generic (PLEG): container finished" podID="26f91aac-4ed2-423a-bbbf-721daee73586" containerID="24dad9ba4328eba17c38abd37db84b9de53b2762c672b3036accdf596a69c197" exitCode=60 Oct 30 01:00:20.312436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:20.312078 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerDied","Data":"24dad9ba4328eba17c38abd37db84b9de53b2762c672b3036accdf596a69c197"} Oct 30 01:00:20.312436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:20.312138 5548 scope.go:122] "RemoveContainer" containerID="d788b5d7fbf4e59279672667f1d6e1b30e910bfcbd7b3593a9166633966105f9" Oct 30 01:00:20.312536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:20.312513 5548 scope.go:122] "RemoveContainer" containerID="24dad9ba4328eba17c38abd37db84b9de53b2762c672b3036accdf596a69c197" Oct 30 01:00:20.312783 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:20.312763 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8_odh-ai-gateway-infra(26f91aac-4ed2-423a-bbbf-721daee73586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" Oct 30 01:00:21.257449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.257411 5548 scope.go:122] "RemoveContainer" containerID="f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2" Oct 30 01:00:21.320280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.320252 5548 generic.go:356] "Generic (PLEG): container finished" podID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerID="51a1d6b10e9192a916f0925d70ebeee11ec709b1d33cf9790e8b63d56e78b44a" exitCode=60 Oct 30 01:00:21.320778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.320335 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerDied","Data":"51a1d6b10e9192a916f0925d70ebeee11ec709b1d33cf9790e8b63d56e78b44a"} Oct 30 01:00:21.320778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.320397 5548 scope.go:122] "RemoveContainer" containerID="97d3d2d22e0a29f9fc892c3586915feb41e20734dec7eea73969d43a23e05f89" Oct 30 01:00:21.321035 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.320819 5548 scope.go:122] "RemoveContainer" containerID="51a1d6b10e9192a916f0925d70ebeee11ec709b1d33cf9790e8b63d56e78b44a" Oct 30 01:00:21.321184 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:21.321159 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh_odh-ai-gateway-infra(c3cae984-2e6a-44af-b0a9-bf889c20cf15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" Oct 30 01:00:21.363937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.363901 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:21.476441 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.476408 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:00:21.482560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.482535 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.507396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.505172 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:00:21.549487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.549461 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:21.602553 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.602527 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\" (UniqueName: \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\") pod \"26f91aac-4ed2-423a-bbbf-721daee73586\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " Oct 30 01:00:21.602692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.602600 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\") pod \"26f91aac-4ed2-423a-bbbf-721daee73586\" (UID: \"26f91aac-4ed2-423a-bbbf-721daee73586\") " Oct 30 01:00:21.602964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.602942 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca" pod "26f91aac-4ed2-423a-bbbf-721daee73586" (UID: "26f91aac-4ed2-423a-bbbf-721daee73586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:21.603047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603011 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603064 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603238 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603220 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603257 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603314 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603334 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrvp\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603358 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603402 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603611 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603489 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.603668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.603612 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26f91aac-4ed2-423a-bbbf-721daee73586-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.605411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.605390 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb" pod "26f91aac-4ed2-423a-bbbf-721daee73586" (UID: "26f91aac-4ed2-423a-bbbf-721daee73586"). InnerVolumeSpecName "kube-api-access-ktvfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:21.705103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705070 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705131 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705200 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705226 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705264 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705281 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrvp\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705317 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705347 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705434 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705523 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.705622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705597 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvfb\" (UniqueName: \"kubernetes.io/projected/26f91aac-4ed2-423a-bbbf-721daee73586-kube-api-access-ktvfb\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.705823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705806 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.706019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.705998 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.706094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.706075 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.706157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.706093 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.708396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.708372 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.708558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.708543 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.722567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.722540 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.722686 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.722623 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrvp\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:21.844103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:21.844023 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:22.177003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.176973 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:00:22.222727 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:00:22.222698 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8f95d1_bbf1_48c5_a958_1807c805d7d3.slice/crio-4995cf62e2c2453c37e341d8a39aeabb4601cf04d3cb10088ed5a0315a09e271 WatchSource:0}: Error finding container 4995cf62e2c2453c37e341d8a39aeabb4601cf04d3cb10088ed5a0315a09e271: Status 404 returned error can't find the container with id 4995cf62e2c2453c37e341d8a39aeabb4601cf04d3cb10088ed5a0315a09e271 Oct 30 01:00:22.225349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.225306 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:22.225438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.225382 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:22.225438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.225419 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:22.334644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.334606 5548 generic.go:356] "Generic (PLEG): container finished" podID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerID="d753f6ea6a461ebb5dc839875f84f068edfc666852ae1080290570fc3188ae45" exitCode=60 Oct 30 01:00:22.335139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.334685 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" event={"ID":"98ff02b9-9a9c-48de-9288-9da9d9e5548b","Type":"ContainerDied","Data":"d753f6ea6a461ebb5dc839875f84f068edfc666852ae1080290570fc3188ae45"} Oct 30 01:00:22.335139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.334768 5548 scope.go:122] "RemoveContainer" containerID="f25704ea92434b1ce693e2fc1ee7762313445c42d40cf3622f015cddb510c2d2" Oct 30 01:00:22.335139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.335115 5548 scope.go:122] "RemoveContainer" containerID="d753f6ea6a461ebb5dc839875f84f068edfc666852ae1080290570fc3188ae45" Oct 30 01:00:22.335398 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:22.335378 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v_odh-ai-gateway-infra(98ff02b9-9a9c-48de-9288-9da9d9e5548b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" Oct 30 01:00:22.337256 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.337233 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" event={"ID":"ab8f95d1-bbf1-48c5-a958-1807c805d7d3","Type":"ContainerStarted","Data":"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc"} Oct 30 01:00:22.337383 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.337282 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" event={"ID":"ab8f95d1-bbf1-48c5-a958-1807c805d7d3","Type":"ContainerStarted","Data":"4995cf62e2c2453c37e341d8a39aeabb4601cf04d3cb10088ed5a0315a09e271"} Oct 30 01:00:22.338853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.338832 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" Oct 30 01:00:22.338976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.338854 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8" event={"ID":"26f91aac-4ed2-423a-bbbf-721daee73586","Type":"ContainerDied","Data":"e695906a8bd866c762b12db80f1e2750d071d54bb49b3d98168fd179faf4be1d"} Oct 30 01:00:22.351165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.351146 5548 scope.go:122] "RemoveContainer" containerID="24dad9ba4328eba17c38abd37db84b9de53b2762c672b3036accdf596a69c197" Oct 30 01:00:22.364623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.364599 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:22.412728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.412685 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:22.416172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.415753 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dca38d-29888700-nqdv8"] Oct 30 01:00:22.449060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.448807 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podStartSLOduration=1.448786621 podStartE2EDuration="1.448786621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:22.448383628 +0000 UTC m=+1013.290944222" watchObservedRunningTime="2026-10-30 01:00:22.448786621 +0000 UTC m=+1013.291347216" Oct 30 01:00:22.570059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.570034 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:22.616578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.616549 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\" (UniqueName: \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\") pod \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " Oct 30 01:00:22.616763 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.616732 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\") pod \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\" (UID: \"c3cae984-2e6a-44af-b0a9-bf889c20cf15\") " Oct 30 01:00:22.617146 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.617116 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca" pod "c3cae984-2e6a-44af-b0a9-bf889c20cf15" (UID: "c3cae984-2e6a-44af-b0a9-bf889c20cf15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:22.617284 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.617266 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3cae984-2e6a-44af-b0a9-bf889c20cf15-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.619518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.619486 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7" pod "c3cae984-2e6a-44af-b0a9-bf889c20cf15" (UID: "c3cae984-2e6a-44af-b0a9-bf889c20cf15"). InnerVolumeSpecName "kube-api-access-58qp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:22.718384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.718310 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qp7\" (UniqueName: \"kubernetes.io/projected/c3cae984-2e6a-44af-b0a9-bf889c20cf15-kube-api-access-58qp7\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.844459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:22.844424 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:23.264720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.264688 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" path="/var/lib/kubelet/pods/26f91aac-4ed2-423a-bbbf-721daee73586/volumes" Oct 30 01:00:23.346269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.346246 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" Oct 30 01:00:23.346269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.346252 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh" event={"ID":"c3cae984-2e6a-44af-b0a9-bf889c20cf15","Type":"ContainerDied","Data":"f5f1add05bb50773b61306e8b5b3ba3664c2b00b6ad5e276eb39d965e13024f6"} Oct 30 01:00:23.346696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.346310 5548 scope.go:122] "RemoveContainer" containerID="51a1d6b10e9192a916f0925d70ebeee11ec709b1d33cf9790e8b63d56e78b44a" Oct 30 01:00:23.396331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.396282 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:23.414275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.414240 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:23.421167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.421134 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888700-5fpgh"] Oct 30 01:00:23.604853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.604823 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:23.635498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.635459 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\" (UniqueName: \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\") pod \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " Oct 30 01:00:23.636376 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.635714 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\") pod \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\" (UID: \"98ff02b9-9a9c-48de-9288-9da9d9e5548b\") " Oct 30 01:00:23.655359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.652158 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca" pod "98ff02b9-9a9c-48de-9288-9da9d9e5548b" (UID: "98ff02b9-9a9c-48de-9288-9da9d9e5548b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:23.666786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.666066 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc" pod "98ff02b9-9a9c-48de-9288-9da9d9e5548b" (UID: "98ff02b9-9a9c-48de-9288-9da9d9e5548b"). InnerVolumeSpecName "kube-api-access-gbsfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:23.737498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.737459 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsfc\" (UniqueName: \"kubernetes.io/projected/98ff02b9-9a9c-48de-9288-9da9d9e5548b-kube-api-access-gbsfc\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.737498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.737494 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ff02b9-9a9c-48de-9288-9da9d9e5548b-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.845767 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.845683 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:23.845767 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:23.845739 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:24.358126 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.358102 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" Oct 30 01:00:24.358578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.358139 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v" event={"ID":"98ff02b9-9a9c-48de-9288-9da9d9e5548b","Type":"ContainerDied","Data":"202ffa6d5a3021e330564122b259865377caa25723f47f8803220d172933d3a1"} Oct 30 01:00:24.358578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.358195 5548 scope.go:122] "RemoveContainer" containerID="d753f6ea6a461ebb5dc839875f84f068edfc666852ae1080290570fc3188ae45" Oct 30 01:00:24.424835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.424806 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:24.429578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.429548 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888700-jz49v"] Oct 30 01:00:24.844483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.844446 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:24.844654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:24.844506 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:25.264659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:25.264624 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" path="/var/lib/kubelet/pods/98ff02b9-9a9c-48de-9288-9da9d9e5548b/volumes" Oct 30 01:00:25.265043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:25.265029 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" path="/var/lib/kubelet/pods/c3cae984-2e6a-44af-b0a9-bf889c20cf15/volumes" Oct 30 01:00:25.432909 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:25.432101 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:00:25.845338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:25.845304 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:25.845528 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:25.845357 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:26.845224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:26.845191 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:26.845603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:26.845248 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:27.845057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:27.845021 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.845373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:27.845078 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:28.844579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:28.844540 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:00:28.844763 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:28.844589 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:00:29.849569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:29.849487 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:29.850020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:29.849893 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:29.850667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:29.850647 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:00:36.792164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792129 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:00:36.792668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792654 5548 cpu_manager.go:414] "containerMap: removing container" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.792720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792669 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.792720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792679 5548 cpu_manager.go:414] "containerMap: removing container" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.792720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792684 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.792720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792706 5548 cpu_manager.go:414] "containerMap: removing container" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792713 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792730 5548 cpu_manager.go:414] "containerMap: removing container" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792735 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792743 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792747 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792757 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792761 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792772 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792776 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792850 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792870 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792879 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792884 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792889 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792896 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cae984-2e6a-44af-b0a9-bf889c20cf15" containerName="cleanup" Oct 30 01:00:36.792902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.792901 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:00:36.798196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.798178 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.818114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.818085 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:00:36.829957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.829926 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.829971 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830006 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830022 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzlw\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830052 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830087 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830177 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830253 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.830388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.830309 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931557 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931595 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931645 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931671 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzlw\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931715 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931769 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931800 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.931919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931912 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.932133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.931994 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.932133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.932018 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.932133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.932081 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.932416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.932340 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.932476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.932419 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.934617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.934596 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.935185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.935163 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.951653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.951612 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:36.951790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:36.951719 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzlw\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:37.111691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:37.111613 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:37.497886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:37.496846 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:00:37.526017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:37.516773 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:37.526017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:37.516913 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:37.526017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:37.516990 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:00:38.569885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:38.553155 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" event={"ID":"de158740-04aa-40fd-ad73-2f1917a9525c","Type":"ContainerStarted","Data":"5ad60e5eedadb8cf274890b88069a53ede130aa0d8e7cb05b63c5897f54288d3"} Oct 30 01:00:38.569885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:38.553224 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" event={"ID":"de158740-04aa-40fd-ad73-2f1917a9525c","Type":"ContainerStarted","Data":"ad5303bb01d8625a8b83705d4ab96c8e813dde7c42c98912d338dbe72f6c6d8e"} Oct 30 01:00:38.642885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:38.631635 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podStartSLOduration=2.631611248 podStartE2EDuration="2.631611248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:38.617426031 +0000 UTC m=+1029.459986637" watchObservedRunningTime="2026-10-30 01:00:38.631611248 +0000 UTC m=+1029.474171869" Oct 30 01:00:39.116004 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:39.115944 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:39.116602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:39.116553 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:39.116715 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:39.116635 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:39.429886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:39.422837 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:00:40.215472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.215436 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:40.218690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.218653 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:40.346887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.307897 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.413889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.380035 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.492934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.439940 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.492934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.486958 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.493174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.492978 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:00:40.509073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.509033 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.587685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:40.587644 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:00:41.115331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.115291 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:41.115671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.115637 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:41.224372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.224322 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.424467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.424418 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:00:41.500886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.497565 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:00:41.506111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.506070 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:00:41.517217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.517186 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:00:41.518301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.506346 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:00:41.520278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.520250 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:00:41.520980 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.520958 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:00:41.625886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.596697 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.676960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.651706 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:00:41.995667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:41.995537 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:00:42.276500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.275325 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:42.276500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.275390 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:42.437088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.437028 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.526763 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.525965 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.526763 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.526420 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.533883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.527683 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.533883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.528782 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:00:42.655884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:42.651960 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:43.034102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.034054 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:43.059446 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.059415 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:00:43.124349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.124317 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:43.124663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.124610 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:43.460453 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.445990 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:00:43.530171 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.529709 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:00:43.572122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.572066 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:43.573904 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.573871 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:43.665884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.658934 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:43.846464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.846383 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:00:43.846738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.846720 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:00:43.849621 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.849595 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:00:43.862080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.862040 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:00:43.865973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.865951 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:00:43.894643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:43.894619 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:00:44.112750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:44.112667 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:44.112750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:44.112733 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:45.112304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:45.112265 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:45.112669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:45.112327 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:46.112828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:46.112799 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:46.113350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:46.112853 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:47.112133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:47.112093 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:47.112322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:47.112146 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:48.112183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:48.112138 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:00:48.112663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:48.112209 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 01:00:49.117389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:49.117357 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:49.117888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:49.117654 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:49.118525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:00:49.118502 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:00:55.427658 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:00:55.427617 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:01:04.878006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:04.877954 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:01:04.879219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:04.878735 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:04.880465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:04.878773 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.441031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.440982 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.585011 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.584965 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.858056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.857954 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:01:05.883052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.883002 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.884498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.884478 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:05.885348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.885306 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.891896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.885492 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.891896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.887007 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:01:05.891896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.887056 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:05.997408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:05.997365 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.204936 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.205785 5548 cpu_manager.go:414] "containerMap: removing container" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.205803 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.205846 5548 cpu_manager.go:414] "containerMap: removing container" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.205853 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.206030 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f91aac-4ed2-423a-bbbf-721daee73586" containerName="cleanup" Oct 30 01:01:06.218898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.206048 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ff02b9-9a9c-48de-9288-9da9d9e5548b" containerName="cleanup" Oct 30 01:01:06.444728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.444677 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.521451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.521369 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:01:06.521849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.521830 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:06.522086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.522070 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:01:06.522263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.522249 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:06.522428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.522415 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:01:06.529897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.529269 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.616871 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.616806 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:01:06.627136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.627110 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:06.695597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.695553 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.695789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.695690 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.695789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.695738 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7sd6\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.773941 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.774017 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.774112 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.774155 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.774457 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.778947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.774499 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.903897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.902970 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.922186 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.922144 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.922434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.922415 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.922669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.922653 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.922873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.922838 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.923011 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.922993 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7sd6\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.923179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.923165 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.923303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.923287 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.923440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.923425 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.923566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.923551 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.924197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.924173 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.924631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.924602 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.927156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.927135 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:01:06.952886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.927686 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.953171 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.929073 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:06.953270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.929675 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.000369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:06.983544 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7sd6\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.039943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.022149 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.229889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.225849 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.353299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.353198 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.368275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.367168 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.449012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.448963 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.562572 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.562539 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:07.635964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.628366 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.924008 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:07.923959 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.349942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:08.348977 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.420747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:08.420681 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.450644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:08.450605 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.668884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:08.659475 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.937935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:08.932955 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:09.365039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.353077 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:09.377114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.371039 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:09.443697 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.434975 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:01:09.468224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.464930 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:01:09.473118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.472884 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:01:09.485098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.479805 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:09.485098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.479898 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:09.501081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.500679 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:09.501081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.500776 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:09.501081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.500825 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:09.509457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.509433 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:09.524427 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.511128 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:09.844805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.844779 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:01:09.858254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.857516 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:01:09.867903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.867847 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" event={"ID":"80783643-0bcf-4a15-9da9-90e679b2cd60","Type":"ContainerStarted","Data":"65e97a2e9ec41ceb9d46dc8b5b6da1ff3dee4bb33baef421c70b615673465213"} Oct 30 01:01:09.869162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:09.869133 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:10.897387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:10.897348 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" event={"ID":"80783643-0bcf-4a15-9da9-90e679b2cd60","Type":"ContainerStarted","Data":"43c727224ac96aacdbf838a8c69786503e8297f635178e0349c8b832737adf4c"} Oct 30 01:01:10.954972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:10.954917 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podStartSLOduration=4.954901238 podStartE2EDuration="4.954901238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:10.943534268 +0000 UTC m=+1061.786094852" watchObservedRunningTime="2026-10-30 01:01:10.954901238 +0000 UTC m=+1061.797461825" Oct 30 01:01:11.573502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:11.573465 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:11.573784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:11.573755 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:11.574231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:11.574209 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:12.592357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:12.592321 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:12.594073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:12.594019 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:13.592323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:13.592276 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:13.592483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:13.592352 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:14.392888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.390590 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw"] Oct 30 01:01:14.401145 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.401119 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw"] Oct 30 01:01:14.401452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.401255 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.414104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.414081 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:01:14.414205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.414168 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:01:14.449456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.449423 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.449622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.449569 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.449663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.449635 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt75t\" (UniqueName: \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.553638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.551751 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.553638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.551845 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.553638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.551919 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt75t\" (UniqueName: \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.553638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.553367 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.563154 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.563111 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.585970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.585485 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:14.585970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.585561 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:14.586942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.586879 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt75t\" (UniqueName: \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\") pod \"maas-api-revoke-sub-872946f5-hlqlw\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:14.718936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:14.718836 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:15.049736 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.049707 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw"] Oct 30 01:01:15.051121 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:01:15.051088 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2a19cb_dd21_49d3_8047_a0630e150928.slice/crio-f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed WatchSource:0}: Error finding container f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed: Status 404 returned error can't find the container with id f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed Oct 30 01:01:15.587636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.587144 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:15.587636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.587202 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:15.941664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.941625 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" event={"ID":"1c2a19cb-dd21-49d3-8047-a0630e150928","Type":"ContainerStarted","Data":"7941f7d5992da1f5827779c8f1489c3b69bf6b854fb0a1f254ce29d620447ebb"} Oct 30 01:01:15.942173 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.941685 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" event={"ID":"1c2a19cb-dd21-49d3-8047-a0630e150928","Type":"ContainerStarted","Data":"f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed"} Oct 30 01:01:15.977136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:15.977087 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" podStartSLOduration=1.97707367 podStartE2EDuration="1.97707367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:15.975061445 +0000 UTC m=+1066.817622039" watchObservedRunningTime="2026-10-30 01:01:15.97707367 +0000 UTC m=+1066.819634282" Oct 30 01:01:16.571748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:16.571715 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:01:16.571957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:16.571765 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:01:16.952472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:16.952432 5548 generic.go:356] "Generic (PLEG): container finished" podID="1c2a19cb-dd21-49d3-8047-a0630e150928" containerID="7941f7d5992da1f5827779c8f1489c3b69bf6b854fb0a1f254ce29d620447ebb" exitCode=0 Oct 30 01:01:16.953062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:16.952511 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" event={"ID":"1c2a19cb-dd21-49d3-8047-a0630e150928","Type":"ContainerDied","Data":"7941f7d5992da1f5827779c8f1489c3b69bf6b854fb0a1f254ce29d620447ebb"} Oct 30 01:01:17.576395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:17.576366 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:17.576801 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:17.576753 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:17.577453 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:17.577437 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:01:18.130382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.130361 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:18.217346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.217260 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\") pod \"1c2a19cb-dd21-49d3-8047-a0630e150928\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " Oct 30 01:01:18.217517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.217412 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\" (UniqueName: \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\") pod \"1c2a19cb-dd21-49d3-8047-a0630e150928\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " Oct 30 01:01:18.217633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.217609 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\") pod \"1c2a19cb-dd21-49d3-8047-a0630e150928\" (UID: \"1c2a19cb-dd21-49d3-8047-a0630e150928\") " Oct 30 01:01:18.218036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.218004 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca" pod "1c2a19cb-dd21-49d3-8047-a0630e150928" (UID: "1c2a19cb-dd21-49d3-8047-a0630e150928"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:18.218520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.218496 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:01:18.219850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.219824 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca" pod "1c2a19cb-dd21-49d3-8047-a0630e150928" (UID: "1c2a19cb-dd21-49d3-8047-a0630e150928"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:18.220255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.220231 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t" pod "1c2a19cb-dd21-49d3-8047-a0630e150928" (UID: "1c2a19cb-dd21-49d3-8047-a0630e150928"). InnerVolumeSpecName "kube-api-access-kt75t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:18.319853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.319821 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2a19cb-dd21-49d3-8047-a0630e150928-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:01:18.319853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.319849 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt75t\" (UniqueName: \"kubernetes.io/projected/1c2a19cb-dd21-49d3-8047-a0630e150928-kube-api-access-kt75t\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:01:18.967342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.967308 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" event={"ID":"1c2a19cb-dd21-49d3-8047-a0630e150928","Type":"ContainerDied","Data":"f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed"} Oct 30 01:01:18.967543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.967347 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw" Oct 30 01:01:18.967543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:18.967371 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48b497873b4b14295799937147ccded9ec1ffef69949fb7ee01bc4b14a198ed" Oct 30 01:01:25.429453 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:25.429415 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:01:31.217010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.216955 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.369420 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.366970 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.455020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.454962 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.518080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.517961 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.518990 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.518956 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.881185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.881079 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.942016 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:31.941964 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.337568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.337527 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:32.338213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.338179 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.374016 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.373976 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:01:32.374672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.374648 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:01:32.497888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.468974 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:01:32.497888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.471323 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:01:32.523887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.521653 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.550918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.540326 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:32.550918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.541139 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.550918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.541239 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.550918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.541612 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:32.932908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.932877 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:32.984888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.984065 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:01:32.984888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.984480 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.010888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.989548 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:01:33.010888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.989629 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:33.010888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:32.989668 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:33.184969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.181015 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:33.192887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.188941 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.385986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.376053 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.484944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.473964 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:01:33.584033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.583991 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:33.584350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.584318 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.632387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.632346 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:33.632666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.632633 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.994331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.993600 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.020903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.995223 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.020903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:33.995289 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:01:34.391115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:34.391010 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.502887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:34.498020 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:01:34.755698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:34.738980 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 01:01:35.024851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:34.975900 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:01:35.024851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.000941 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:01:35.024851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.008874 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:35.024851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.009534 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:35.024851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.019941 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 01:01:35.182394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.182010 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:35.182394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.182070 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:35.524499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.524442 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:35.750896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:35.724126 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:01:36.054064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.054015 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:36.070768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.054393 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:36.193010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.192953 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:01:36.258244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.195342 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:01:36.259121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.196241 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:36.333688 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:36.326083 5548 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.002s" Oct 30 01:01:36.334268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.334249 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:01:36.334810 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.334793 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:36.336014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.336000 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:01:36.336196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.336175 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:01:36.338999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.338985 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:01:36.340462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.340439 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:01:36.341558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:36.341541 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:01:41.972684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.972608 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8762-9958fd"] Oct 30 01:01:41.973333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.973315 5548 cpu_manager.go:414] "containerMap: removing container" podUID="1c2a19cb-dd21-49d3-8047-a0630e150928" containerName="revoke-keys" Oct 30 01:01:41.973378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.973335 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2a19cb-dd21-49d3-8047-a0630e150928" containerName="revoke-keys" Oct 30 01:01:41.973443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.973433 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2a19cb-dd21-49d3-8047-a0630e150928" containerName="revoke-keys" Oct 30 01:01:41.976900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.976848 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:41.991186 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.991163 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8762-9958fd"] Oct 30 01:01:41.997666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:41.997631 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:01:42.070529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.070488 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzsj\" (UniqueName: \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\") pod \"test-curl-8762-9958fd\" (UID: \"e73f892f-9cbc-4709-b771-33c5ae59417f\") " pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:42.172260 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.172229 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzsj\" (UniqueName: \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\") pod \"test-curl-8762-9958fd\" (UID: \"e73f892f-9cbc-4709-b771-33c5ae59417f\") " pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:42.191532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.191498 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzsj\" (UniqueName: \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\") pod \"test-curl-8762-9958fd\" (UID: \"e73f892f-9cbc-4709-b771-33c5ae59417f\") " pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:42.288595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.288511 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:42.607960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.607934 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8762-9958fd"] Oct 30 01:01:42.610085 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:01:42.610057 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73f892f_9cbc_4709_b771_33c5ae59417f.slice/crio-b777b443ebae565b01885c7d6985ce70c8c04eff2fe1ad28537dbfc1cbda1113 WatchSource:0}: Error finding container b777b443ebae565b01885c7d6985ce70c8c04eff2fe1ad28537dbfc1cbda1113: Status 404 returned error can't find the container with id b777b443ebae565b01885c7d6985ce70c8c04eff2fe1ad28537dbfc1cbda1113 Oct 30 01:01:42.612134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:42.612116 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:01:43.252604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:43.252567 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8762-9958fd" event={"ID":"e73f892f-9cbc-4709-b771-33c5ae59417f","Type":"ContainerStarted","Data":"cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0"} Oct 30 01:01:43.252986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:43.252618 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8762-9958fd" event={"ID":"e73f892f-9cbc-4709-b771-33c5ae59417f","Type":"ContainerStarted","Data":"b777b443ebae565b01885c7d6985ce70c8c04eff2fe1ad28537dbfc1cbda1113"} Oct 30 01:01:43.285689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:43.285645 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8762-9958fd" podStartSLOduration=1.763388823 podStartE2EDuration="2.285631247s" totalImagesPullingTime="522.242424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:43.283995973 +0000 UTC m=+1094.126556566" watchObservedRunningTime="2026-10-30 01:01:43.285631247 +0000 UTC m=+1094.128191837" Oct 30 01:01:44.109112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.109080 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8762-9958fd"] Oct 30 01:01:44.115421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.112840 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8762-9958fd"] Oct 30 01:01:44.292348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.292314 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8762-187c93"] Oct 30 01:01:44.292939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.292924 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" containerName="test-curl-8762-9958fd" Oct 30 01:01:44.292991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.292940 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" containerName="test-curl-8762-9958fd" Oct 30 01:01:44.293056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.293047 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" containerName="test-curl-8762-9958fd" Oct 30 01:01:44.296647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.296633 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:44.311416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.311395 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8762-187c93"] Oct 30 01:01:44.400251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.400220 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74xj\" (UniqueName: \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\") pod \"test-curl-8762-187c93\" (UID: \"c68cf818-5aee-4ed0-84be-10e0c071561c\") " pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:44.501591 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.501562 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74xj\" (UniqueName: \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\") pod \"test-curl-8762-187c93\" (UID: \"c68cf818-5aee-4ed0-84be-10e0c071561c\") " pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:44.518250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.518228 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74xj\" (UniqueName: \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\") pod \"test-curl-8762-187c93\" (UID: \"c68cf818-5aee-4ed0-84be-10e0c071561c\") " pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:44.609704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.609671 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:44.914795 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:44.914767 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8762-187c93"] Oct 30 01:01:44.915513 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:01:44.915481 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc68cf818_5aee_4ed0_84be_10e0c071561c.slice/crio-440e9a1502d1a0dbe583f44fad2ce5ff7d14f0378bb15e9dcca2920c1eb0b1f3 WatchSource:0}: Error finding container 440e9a1502d1a0dbe583f44fad2ce5ff7d14f0378bb15e9dcca2920c1eb0b1f3: Status 404 returned error can't find the container with id 440e9a1502d1a0dbe583f44fad2ce5ff7d14f0378bb15e9dcca2920c1eb0b1f3 Oct 30 01:01:45.268244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:45.268161 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8762-187c93" event={"ID":"c68cf818-5aee-4ed0-84be-10e0c071561c","Type":"ContainerStarted","Data":"43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568"} Oct 30 01:01:45.268244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:45.268207 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8762-187c93" event={"ID":"c68cf818-5aee-4ed0-84be-10e0c071561c","Type":"ContainerStarted","Data":"440e9a1502d1a0dbe583f44fad2ce5ff7d14f0378bb15e9dcca2920c1eb0b1f3"} Oct 30 01:01:45.268470 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:45.268237 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8762-9958fd" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" containerName="test-curl-8762-9958fd" containerID="cri-o://cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0" gracePeriod=2 Oct 30 01:01:45.301845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:45.301790 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8762-187c93" podStartSLOduration=1.301777462 podStartE2EDuration="1.301777462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:45.29862576 +0000 UTC m=+1096.141186350" watchObservedRunningTime="2026-10-30 01:01:45.301777462 +0000 UTC m=+1096.144338055" Oct 30 01:01:46.133578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:46.133547 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8762-187c93"] Oct 30 01:01:46.138550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:46.138526 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8762-187c93"] Oct 30 01:01:47.281908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:47.281840 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8762-187c93" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" containerName="test-curl-8762-187c93" containerID="cri-o://43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568" gracePeriod=2 Oct 30 01:01:47.460458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:47.460432 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:47.542191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:47.542119 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\" (UniqueName: \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\") pod \"e73f892f-9cbc-4709-b771-33c5ae59417f\" (UID: \"e73f892f-9cbc-4709-b771-33c5ae59417f\") " Oct 30 01:01:47.544525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:47.544490 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj" pod "e73f892f-9cbc-4709-b771-33c5ae59417f" (UID: "e73f892f-9cbc-4709-b771-33c5ae59417f"). InnerVolumeSpecName "kube-api-access-7qzsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:47.643280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:47.643241 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzsj\" (UniqueName: \"kubernetes.io/projected/e73f892f-9cbc-4709-b771-33c5ae59417f-kube-api-access-7qzsj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:01:48.302852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:48.302821 5548 generic.go:356] "Generic (PLEG): container finished" podID="e73f892f-9cbc-4709-b771-33c5ae59417f" containerID="cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0" exitCode=137 Oct 30 01:01:48.303392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:48.303048 5548 scope.go:122] "RemoveContainer" containerID="cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0" Oct 30 01:01:48.303392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:48.303169 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:48.321802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:48.321781 5548 scope.go:122] "RemoveContainer" containerID="cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0" Oct 30 01:01:48.322116 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:48.322093 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0\": container with ID starting with cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0 not found: ID does not exist" containerID="cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0" Oct 30 01:01:48.322214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:48.322128 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0"} err="failed to get container status \"cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0\": rpc error: code = NotFound desc = could not find container \"cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0\": container with ID starting with cfd2c7b280f224cf4b98199af027fcf1e72855847cb350b66709a05faff0f9b0 not found: ID does not exist" Oct 30 01:01:48.326501 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:48.326473 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8762-9958fd\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:49.278118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:49.278080 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" path="/var/lib/kubelet/pods/e73f892f-9cbc-4709-b771-33c5ae59417f/volumes" Oct 30 01:01:49.289387 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:49.288994 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8762-9958fd\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="e73f892f-9cbc-4709-b771-33c5ae59417f" pod="opendatahub/test-curl-8762-9958fd" Oct 30 01:01:49.492505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:49.492474 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:49.575921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:49.575819 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\" (UniqueName: \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\") pod \"c68cf818-5aee-4ed0-84be-10e0c071561c\" (UID: \"c68cf818-5aee-4ed0-84be-10e0c071561c\") " Oct 30 01:01:49.578111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:49.578082 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj" pod "c68cf818-5aee-4ed0-84be-10e0c071561c" (UID: "c68cf818-5aee-4ed0-84be-10e0c071561c"). InnerVolumeSpecName "kube-api-access-k74xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:49.677994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:49.677960 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k74xj\" (UniqueName: \"kubernetes.io/projected/c68cf818-5aee-4ed0-84be-10e0c071561c-kube-api-access-k74xj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:01:50.319302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:50.319268 5548 generic.go:356] "Generic (PLEG): container finished" podID="c68cf818-5aee-4ed0-84be-10e0c071561c" containerID="43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568" exitCode=137 Oct 30 01:01:50.319493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:50.319318 5548 scope.go:122] "RemoveContainer" containerID="43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568" Oct 30 01:01:50.319493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:50.319317 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:50.330650 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:50.330631 5548 scope.go:122] "RemoveContainer" containerID="43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568" Oct 30 01:01:50.331043 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:50.331022 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568\": container with ID starting with 43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568 not found: ID does not exist" containerID="43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568" Oct 30 01:01:50.331129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:50.331051 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568"} err="failed to get container status \"43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568\": rpc error: code = NotFound desc = could not find container \"43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568\": container with ID starting with 43b3eb5751311874d184149afd6c6340d8d90afef48e09b70c93f1116487a568 not found: ID does not exist" Oct 30 01:01:50.339798 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:50.339767 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8762-187c93\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:51.055377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.055344 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:01:51.055935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.055913 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" containerName="test-curl-8762-187c93" Oct 30 01:01:51.055935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.055931 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" containerName="test-curl-8762-187c93" Oct 30 01:01:51.056160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.056129 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" containerName="test-curl-8762-187c93" Oct 30 01:01:51.060899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.060847 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.088513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.088486 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:01:51.109378 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:51.109146 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8762-187c93\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" pod="opendatahub/test-curl-8762-187c93" Oct 30 01:01:51.197765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.197731 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.197939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.197777 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.197939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.197891 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198019 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198051 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwss\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198082 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198182 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198281 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.198345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.198333 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.267416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.267376 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68cf818-5aee-4ed0-84be-10e0c071561c" path="/var/lib/kubelet/pods/c68cf818-5aee-4ed0-84be-10e0c071561c/volumes" Oct 30 01:01:51.299217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299182 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299223 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwss\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299353 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299399 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299463 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299537 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299620 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299665 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299687 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299723 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299767 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.299976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.299942 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.300055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.300031 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.300327 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.300304 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.302271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.302248 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.302395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.302380 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.318134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.318045 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.318276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.318180 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwss\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.375728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.375680 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:51.736083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.736052 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:01:51.737783 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:01:51.737756 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8a07ae_22a0_4165_af30_b9c46da08a2e.slice/crio-e357756136c40c1fe0292b9e01f37c3ca4aab48950a32604d5b69ebf6568fa01 WatchSource:0}: Error finding container e357756136c40c1fe0292b9e01f37c3ca4aab48950a32604d5b69ebf6568fa01: Status 404 returned error can't find the container with id e357756136c40c1fe0292b9e01f37c3ca4aab48950a32604d5b69ebf6568fa01 Oct 30 01:01:51.740524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.740484 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:51.740610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.740595 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:51.740651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:51.740637 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:52.339181 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.339139 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" event={"ID":"dc8a07ae-22a0-4165-af30-b9c46da08a2e","Type":"ContainerStarted","Data":"ae016cb73d940802abfe5a989006f9d4faeea3253d6db6ffff338e8c1bb067c4"} Oct 30 01:01:52.339703 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.339206 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" event={"ID":"dc8a07ae-22a0-4165-af30-b9c46da08a2e","Type":"ContainerStarted","Data":"e357756136c40c1fe0292b9e01f37c3ca4aab48950a32604d5b69ebf6568fa01"} Oct 30 01:01:52.376570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.376173 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:01:52.378345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.378246 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:52.378345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.378324 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:52.391390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:52.391347 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podStartSLOduration=1.391333258 podStartE2EDuration="1.391333258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:52.387145454 +0000 UTC m=+1103.229706074" watchObservedRunningTime="2026-10-30 01:01:52.391333258 +0000 UTC m=+1103.233893931" Oct 30 01:01:53.380893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:53.376421 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:53.380893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:53.376481 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:54.385886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:54.384950 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:54.385886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:54.385007 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:55.388885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:55.385614 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:55.388885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:55.385690 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:55.429112 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:01:55.429071 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:01:56.300358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.300314 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:01:56.305052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.305030 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318067 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318178 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318235 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318264 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318303 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318356 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318393 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318519 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt4v\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.318585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.318561 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.335350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.334181 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b61ed672-openshift-default-dockercfg-j885f\"" Oct 30 01:01:56.351953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.351922 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:01:56.416912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.412852 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:56.416912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.412935 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.422735 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt4v\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.422786 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.422972 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423033 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423085 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423114 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423155 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423206 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423246 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.423732 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.424673 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.424945 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.425262 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.425747 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.429916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.428048 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.431105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.431083 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.455478 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.455446 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.455887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.455844 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt4v\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\") pod \"e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:56.627939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:56.625100 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:01:57.377396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:57.377362 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:57.377737 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:57.377689 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:58.006464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.006414 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.166948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.049821 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.358934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.341483 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.350024 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.351309 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.445700 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.464350 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.558002 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.509295 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.558108 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.558161 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:01:58.560638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:58.505290 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:01:59.009111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.009007 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.013678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.012569 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:01:59.177691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.177640 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" event={"ID":"4f50c84a-2dcd-4a70-8006-cd842ce95230","Type":"ContainerStarted","Data":"ff0c286c09168ad740cf772015def2c63536f7b48e0d7b32a8072de296079708"} Oct 30 01:01:59.452887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.452443 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:01:59.497884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.488130 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:01:59.497884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.488517 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:01:59.521341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.514327 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:01:59.566936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.559096 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.593888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.593607 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:01:59.607275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:01:59.594408 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:02:00.114885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.114404 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.331142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.331043 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.464896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.439455 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:02:00.587665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.587624 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:02:00.587961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.540403 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 01:02:00.588772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.462094 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.623110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.623060 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.677667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.593930 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:02:00.678172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.627951 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:02:00.678524 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:00.593878 5548 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.294s" Oct 30 01:02:00.678793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.678772 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:02:00.678988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.678973 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:02:00.679150 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.679124 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:02:00.679286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.679274 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:02:00.679434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.679421 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:02:00.702107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.702063 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" event={"ID":"4f50c84a-2dcd-4a70-8006-cd842ce95230","Type":"ContainerStarted","Data":"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa"} Oct 30 01:02:00.723124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.723038 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:00.723374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.723347 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.826894 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.773228 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:01.016031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.908928 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.016031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.904442 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:01.016031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:00.909015 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.016031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.001298 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:01.016031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.001353 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.085890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.068195 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:01.085890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.068257 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.199068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.199019 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.418889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.395100 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:02:01.418889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.395162 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:02:01.590007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.589946 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.679940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.660408 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:02:01.679940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.667285 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.682456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.682419 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.709888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.691098 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:02:01.709888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.691539 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:02:01.709888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.692684 5548 patch_prober.go:29] interesting pod/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:02:01.709888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.692738 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:02:01.799886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:01.778115 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" podStartSLOduration=5.778092319 podStartE2EDuration="5.778092319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:01.422339835 +0000 UTC m=+1112.264900429" watchObservedRunningTime="2026-10-30 01:02:01.778092319 +0000 UTC m=+1112.620652913" Oct 30 01:02:02.228123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.228076 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:02.381302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.381262 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:02:02.381611 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.381580 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:02:02.551208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.551101 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:02.609579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.596738 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:02.650836 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.650806 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:02:02.713178 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.713117 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 01:02:02.713968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.713926 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:02:02.733706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.714811 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 01:02:02.734959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.713567 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 01:02:02.998605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:02.998571 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:02:03.008133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.008103 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:02:03.229177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.229130 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.432897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.421915 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 01:02:03.432897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.421972 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 01:02:03.555197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.555160 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:03.555489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.555461 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.555954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.555921 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:03.556119 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.556101 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.612853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.612809 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.638994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.638944 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:03.644964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.644923 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.715073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.714968 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 30 01:02:03.722018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.721971 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.734743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.734699 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:03.790049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:03.790004 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:04.154888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.147269 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:02:04.165197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.165167 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:02:04.167477 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.167456 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:02:04.202633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.202607 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:02:04.243205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.243177 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:02:04.243828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.243813 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:02:04.501961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.501879 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:02:04.502275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.502252 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:02:04.503142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:04.503120 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:02:08.465213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.465125 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:08.469692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.469634 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.489634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.489610 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:08.494877 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.494842 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0660f91f-serving-cert\"" Oct 30 01:02:08.544461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.544430 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5bv5\" (UniqueName: \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.544637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.544602 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.544691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.544655 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.544736 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.544689 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.646261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.646221 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5bv5\" (UniqueName: \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.646477 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.646391 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.646477 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.646445 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.646738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.646514 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.647381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.647358 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.649368 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.649343 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.649476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.649425 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.664098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.664076 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5bv5\" (UniqueName: \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\") pod \"maas-api-e2e-ait-0660f91f-5987944746-hl46f\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:08.783800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:08.783709 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:09.108368 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:09.108337 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:09.109302 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:02:09.109272 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7d5d35_fe93_4ad7_b9f8_6878c59d63b0.slice/crio-3a0614e947ae91cafe42ef85f7e0a72f939e337146344a8e34ed37bb049b096f WatchSource:0}: Error finding container 3a0614e947ae91cafe42ef85f7e0a72f939e337146344a8e34ed37bb049b096f: Status 404 returned error can't find the container with id 3a0614e947ae91cafe42ef85f7e0a72f939e337146344a8e34ed37bb049b096f Oct 30 01:02:10.066236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:10.066176 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" event={"ID":"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0","Type":"ContainerStarted","Data":"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c"} Oct 30 01:02:10.067105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:10.066398 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:10.067105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:10.066436 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" event={"ID":"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0","Type":"ContainerStarted","Data":"3a0614e947ae91cafe42ef85f7e0a72f939e337146344a8e34ed37bb049b096f"} Oct 30 01:02:10.103985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:10.103941 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" podStartSLOduration=1.738746235 podStartE2EDuration="2.10392813s" totalImagesPullingTime="365.181895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:10.099497024 +0000 UTC m=+1120.942057619" watchObservedRunningTime="2026-10-30 01:02:10.10392813 +0000 UTC m=+1120.946488724" Oct 30 01:02:13.829414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.829372 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:13.833947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.833926 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:13.859242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.859215 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8b58192f-serving-cert\"" Oct 30 01:02:13.863989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.863961 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:13.919827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.919793 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lt6\" (UniqueName: \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:13.920073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.919834 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:13.920073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.919969 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:13.920212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:13.920069 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.022070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.022036 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lt6\" (UniqueName: \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.022070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.022074 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.022322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.022112 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.022322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.022152 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.022956 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.022853 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.024698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.024677 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.024920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.024892 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.039609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.039581 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lt6\" (UniqueName: \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\") pod \"maas-api-e2e-derive-8b58192f-6f59b9694-klkvb\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.150815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.150786 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:14.462167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:14.462141 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:14.462874 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:02:14.462830 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ae6b9d_ca10_4b3c_9d25_4217e5cafe10.slice/crio-c1c5c1c29a58397acd649b6d5f57b105bd716ea2ee7315da3cf52177ab63e681 WatchSource:0}: Error finding container c1c5c1c29a58397acd649b6d5f57b105bd716ea2ee7315da3cf52177ab63e681: Status 404 returned error can't find the container with id c1c5c1c29a58397acd649b6d5f57b105bd716ea2ee7315da3cf52177ab63e681 Oct 30 01:02:15.102683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:15.102596 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" event={"ID":"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10","Type":"ContainerStarted","Data":"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267"} Oct 30 01:02:15.102683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:15.102666 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:15.103159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:15.102701 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" event={"ID":"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10","Type":"ContainerStarted","Data":"c1c5c1c29a58397acd649b6d5f57b105bd716ea2ee7315da3cf52177ab63e681"} Oct 30 01:02:16.079652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:16.079624 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:16.118396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:16.118338 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" podStartSLOduration=2.744295151 podStartE2EDuration="3.118321609s" totalImagesPullingTime="374.026458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:15.150037728 +0000 UTC m=+1125.992598322" watchObservedRunningTime="2026-10-30 01:02:16.118321609 +0000 UTC m=+1126.960882253" Oct 30 01:02:20.670195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.670161 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:20.678193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.678168 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.692232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.692207 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:20.706891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.704097 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b61ed672-serving-cert\"" Oct 30 01:02:20.733445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.733412 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.733609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.733586 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.733797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.733778 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.733850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.733811 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5nv\" (UniqueName: \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.835072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.835038 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.835240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.835146 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.835240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.835166 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5nv\" (UniqueName: \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.835240 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:20.835185 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-serving-cert: secret "maas-api-e2e-ait-b61ed672-serving-cert" not found Oct 30 01:02:20.835372 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:20.835243 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls podName:dc6c7361-1991-4ba2-bed3-c463252a1b58 nodeName:}" failed. No retries permitted until 2026-10-30 01:02:21.335227087 +0000 UTC m=+1132.177787661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls") pod "maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" (UID: "dc6c7361-1991-4ba2-bed3-c463252a1b58") : secret "maas-api-e2e-ait-b61ed672-serving-cert" not found Oct 30 01:02:20.835372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.835352 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.835792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.835771 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.837672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.837648 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:20.853463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:20.853408 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5nv\" (UniqueName: \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:21.118594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:21.118522 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:21.343003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:21.342969 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:21.345664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:21.345636 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") pod \"maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:21.598302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:21.598258 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:21.936305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:21.936268 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:21.938375 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:02:21.938348 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6c7361_1991_4ba2_bed3_c463252a1b58.slice/crio-159173cc81a0eb7d5b5b34485c1f02d92dfbe8cb88c9ef6104e0e1a5439fcb7e WatchSource:0}: Error finding container 159173cc81a0eb7d5b5b34485c1f02d92dfbe8cb88c9ef6104e0e1a5439fcb7e: Status 404 returned error can't find the container with id 159173cc81a0eb7d5b5b34485c1f02d92dfbe8cb88c9ef6104e0e1a5439fcb7e Oct 30 01:02:22.154724 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:22.154694 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" event={"ID":"dc6c7361-1991-4ba2-bed3-c463252a1b58","Type":"ContainerStarted","Data":"159173cc81a0eb7d5b5b34485c1f02d92dfbe8cb88c9ef6104e0e1a5439fcb7e"} Oct 30 01:02:23.164261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:23.164223 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" event={"ID":"dc6c7361-1991-4ba2-bed3-c463252a1b58","Type":"ContainerStarted","Data":"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307"} Oct 30 01:02:23.164673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:23.164356 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:23.201301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:23.201253 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" podStartSLOduration=2.821898796 podStartE2EDuration="3.201238988s" totalImagesPullingTime="379.340192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:23.197761154 +0000 UTC m=+1134.040321747" watchObservedRunningTime="2026-10-30 01:02:23.201238988 +0000 UTC m=+1134.043799621" Oct 30 01:02:25.427671 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:25.427633 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:02:27.622033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:27.621985 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.686780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:27.686735 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.697987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:27.697948 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.720792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:27.720754 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:29.175894 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:29.175844 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:47.682347 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:47.682311 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:47.682978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:47.682573 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" podUID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" containerName="maas-api" containerID="cri-o://43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c" gracePeriod=30 Oct 30 01:02:47.999680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:47.999650 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:48.031823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.031782 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\") pod \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " Oct 30 01:02:48.032020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.031984 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\" (UniqueName: \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\") pod \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " Oct 30 01:02:48.032254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.032233 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\") pod \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " Oct 30 01:02:48.032371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.032313 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\") pod \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\" (UID: \"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0\") " Oct 30 01:02:48.032911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.032827 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca" pod "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" (UID: "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:48.033098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.033077 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:48.035420 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.035383 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs" pod "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" (UID: "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:48.035566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.035534 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5" pod "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" (UID: "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0"). InnerVolumeSpecName "kube-api-access-h5bv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:48.036649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.036622 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls" pod "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" (UID: "ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:48.134575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.134536 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:48.134575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.134569 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:48.134799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.134584 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5bv5\" (UniqueName: \"kubernetes.io/projected/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0-kube-api-access-h5bv5\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:48.567426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.567388 5548 generic.go:356] "Generic (PLEG): container finished" podID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" containerID="43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c" exitCode=0 Oct 30 01:02:48.567632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.567448 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" Oct 30 01:02:48.567632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.567485 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" event={"ID":"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0","Type":"ContainerDied","Data":"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c"} Oct 30 01:02:48.567632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.567557 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f" event={"ID":"ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0","Type":"ContainerDied","Data":"3a0614e947ae91cafe42ef85f7e0a72f939e337146344a8e34ed37bb049b096f"} Oct 30 01:02:48.567632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.567590 5548 scope.go:122] "RemoveContainer" containerID="43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c" Oct 30 01:02:48.580689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.580670 5548 scope.go:122] "RemoveContainer" containerID="43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c" Oct 30 01:02:48.581063 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:48.581044 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c\": container with ID starting with 43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c not found: ID does not exist" containerID="43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c" Oct 30 01:02:48.581129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.581072 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c"} err="failed to get container status \"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c\": rpc error: code = NotFound desc = could not find container \"43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c\": container with ID starting with 43fdf81d019f32ddc4f9654b967f40f703fab0dc9a8d546a24227488796d4d4c not found: ID does not exist" Oct 30 01:02:48.614535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.614488 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:48.623033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:48.623002 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0660f91f-5987944746-hl46f"] Oct 30 01:02:49.275439 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.275401 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" path="/var/lib/kubelet/pods/ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0/volumes" Oct 30 01:02:49.916174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.916145 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:02:49.916825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.916800 5548 cpu_manager.go:414] "containerMap: removing container" podUID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" containerName="maas-api" Oct 30 01:02:49.916825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.916820 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" containerName="maas-api" Oct 30 01:02:49.917100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.916960 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7d5d35-fe93-4ad7-b9f8-6878c59d63b0" containerName="maas-api" Oct 30 01:02:49.922666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.922636 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.951823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.951790 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:02:49.970214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.970166 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-837d7038-gw-openshift-default-dockercfg-89scj\"" Oct 30 01:02:49.971104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971072 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971141 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971208 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62l4s\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971240 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971335 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971371 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971471 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971539 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:49.971677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:49.971607 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.083757 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.083944 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084174 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084226 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084300 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62l4s\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084337 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084466 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.084567 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.085214 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.088883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.085445 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.117928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.090745 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.117928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.091596 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.117928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.096246 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.117928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.100617 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.147885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.137917 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.147885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.141791 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62l4s\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.147885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.143534 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\") pod \"e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.252947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.252496 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:50.633076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.633042 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:02:50.633624 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:02:50.633594 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3016dc0d_8b19_4596_9ffb_01eec7d685d6.slice/crio-b5b8972ed387b2b56e380b6d3a52ad78d81714e651ed1e636443dc25403a355c WatchSource:0}: Error finding container b5b8972ed387b2b56e380b6d3a52ad78d81714e651ed1e636443dc25403a355c: Status 404 returned error can't find the container with id b5b8972ed387b2b56e380b6d3a52ad78d81714e651ed1e636443dc25403a355c Oct 30 01:02:50.636212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.636176 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:02:50.636302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.636250 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:02:50.636302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:50.636290 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:02:51.608852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:51.608814 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerStarted","Data":"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff"} Oct 30 01:02:51.609031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:51.608893 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerStarted","Data":"b5b8972ed387b2b56e380b6d3a52ad78d81714e651ed1e636443dc25403a355c"} Oct 30 01:02:51.661164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:51.661112 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podStartSLOduration=2.6610994569999997 podStartE2EDuration="2.661099457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:51.657948204 +0000 UTC m=+1162.500508799" watchObservedRunningTime="2026-10-30 01:02:51.661099457 +0000 UTC m=+1162.503660049" Oct 30 01:02:51.793887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:51.793840 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:51.794117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:51.794076 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" podUID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" containerName="maas-api" containerID="cri-o://078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267" gracePeriod=30 Oct 30 01:02:52.116036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.116008 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:52.219115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.219085 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\") pod \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " Oct 30 01:02:52.219321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.219215 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\" (UniqueName: \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\") pod \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " Oct 30 01:02:52.219474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.219456 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\") pod \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " Oct 30 01:02:52.219575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.219502 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\") pod \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\" (UID: \"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10\") " Oct 30 01:02:52.220233 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.220200 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca" pod "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" (UID: "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:52.222248 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.222215 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs" pod "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" (UID: "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:52.222476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.222455 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls" pod "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" (UID: "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:52.222896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.222826 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6" pod "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" (UID: "20ae6b9d-ca10-4b3c-9d25-4217e5cafe10"). InnerVolumeSpecName "kube-api-access-57lt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:52.254030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.254000 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:52.260032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.259998 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:52.322730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.322623 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:52.322730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.322662 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:52.322730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.322681 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:52.322730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.322695 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lt6\" (UniqueName: \"kubernetes.io/projected/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10-kube-api-access-57lt6\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:52.623970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.620794 5548 generic.go:356] "Generic (PLEG): container finished" podID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" containerID="078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267" exitCode=0 Oct 30 01:02:52.623970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.622500 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" Oct 30 01:02:52.626416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.626384 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" event={"ID":"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10","Type":"ContainerDied","Data":"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267"} Oct 30 01:02:52.626571 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.626450 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb" event={"ID":"20ae6b9d-ca10-4b3c-9d25-4217e5cafe10","Type":"ContainerDied","Data":"c1c5c1c29a58397acd649b6d5f57b105bd716ea2ee7315da3cf52177ab63e681"} Oct 30 01:02:52.626571 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.626480 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:52.626571 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.626513 5548 scope.go:122] "RemoveContainer" containerID="078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267" Oct 30 01:02:52.630072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.629959 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:02:52.642085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.642064 5548 scope.go:122] "RemoveContainer" containerID="078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267" Oct 30 01:02:52.642483 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:52.642462 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267\": container with ID starting with 078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267 not found: ID does not exist" containerID="078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267" Oct 30 01:02:52.642568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.642495 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267"} err="failed to get container status \"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267\": rpc error: code = NotFound desc = could not find container \"078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267\": container with ID starting with 078790f15b6373aa217673b8b8e54344b671768593de14aff01618d13e7ed267 not found: ID does not exist" Oct 30 01:02:52.719953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.719920 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:52.725307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:52.725281 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8b58192f-6f59b9694-klkvb"] Oct 30 01:02:53.266467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:53.266440 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" path="/var/lib/kubelet/pods/20ae6b9d-ca10-4b3c-9d25-4217e5cafe10/volumes" Oct 30 01:02:55.506634 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:55.506513 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:02:57.927269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:57.927235 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:57.927732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:57.927454 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" podUID="dc6c7361-1991-4ba2-bed3-c463252a1b58" containerName="maas-api" containerID="cri-o://6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307" gracePeriod=30 Oct 30 01:02:58.536890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.535257 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:58.708942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.708817 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\") pod \"dc6c7361-1991-4ba2-bed3-c463252a1b58\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " Oct 30 01:02:58.709208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.709187 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") pod \"dc6c7361-1991-4ba2-bed3-c463252a1b58\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " Oct 30 01:02:58.709396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.709378 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\") pod \"dc6c7361-1991-4ba2-bed3-c463252a1b58\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " Oct 30 01:02:58.709580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.709562 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\" (UniqueName: \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\") pod \"dc6c7361-1991-4ba2-bed3-c463252a1b58\" (UID: \"dc6c7361-1991-4ba2-bed3-c463252a1b58\") " Oct 30 01:02:58.710597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.709619 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca" pod "dc6c7361-1991-4ba2-bed3-c463252a1b58" (UID: "dc6c7361-1991-4ba2-bed3-c463252a1b58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:58.716486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.714517 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs" pod "dc6c7361-1991-4ba2-bed3-c463252a1b58" (UID: "dc6c7361-1991-4ba2-bed3-c463252a1b58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:58.717978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.717914 5548 generic.go:356] "Generic (PLEG): container finished" podID="dc6c7361-1991-4ba2-bed3-c463252a1b58" containerID="6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307" exitCode=0 Oct 30 01:02:58.718166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.718098 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" event={"ID":"dc6c7361-1991-4ba2-bed3-c463252a1b58","Type":"ContainerDied","Data":"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307"} Oct 30 01:02:58.718293 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.718276 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" event={"ID":"dc6c7361-1991-4ba2-bed3-c463252a1b58","Type":"ContainerDied","Data":"159173cc81a0eb7d5b5b34485c1f02d92dfbe8cb88c9ef6104e0e1a5439fcb7e"} Oct 30 01:02:58.718373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.718342 5548 scope.go:122] "RemoveContainer" containerID="6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307" Oct 30 01:02:58.720518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.718591 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt" Oct 30 01:02:58.720518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.720307 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls" pod "dc6c7361-1991-4ba2-bed3-c463252a1b58" (UID: "dc6c7361-1991-4ba2-bed3-c463252a1b58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:58.720853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.720830 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv" pod "dc6c7361-1991-4ba2-bed3-c463252a1b58" (UID: "dc6c7361-1991-4ba2-bed3-c463252a1b58"). InnerVolumeSpecName "kube-api-access-2r5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:58.733744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.733726 5548 scope.go:122] "RemoveContainer" containerID="6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307" Oct 30 01:02:58.734081 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:02:58.734054 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307\": container with ID starting with 6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307 not found: ID does not exist" containerID="6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307" Oct 30 01:02:58.734212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.734086 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307"} err="failed to get container status \"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307\": rpc error: code = NotFound desc = could not find container \"6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307\": container with ID starting with 6ceadedcc0241270dc77b84708a2d7e532529ff4d4e9929b8fe2c195084f6307 not found: ID does not exist" Oct 30 01:02:58.812454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.812355 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:58.812454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.812397 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r5nv\" (UniqueName: \"kubernetes.io/projected/dc6c7361-1991-4ba2-bed3-c463252a1b58-kube-api-access-2r5nv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:58.812454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.812414 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c7361-1991-4ba2-bed3-c463252a1b58-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:58.812454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:58.812427 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c7361-1991-4ba2-bed3-c463252a1b58-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:02:59.062888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:59.062839 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:59.098247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:59.098211 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b61ed672-7dd968dc6f-dzpxt"] Oct 30 01:02:59.265044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:02:59.265012 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6c7361-1991-4ba2-bed3-c463252a1b58" path="/var/lib/kubelet/pods/dc6c7361-1991-4ba2-bed3-c463252a1b58/volumes" Oct 30 01:03:00.161649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:00.161595 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:03:00.162569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:00.162537 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" containerID="cri-o://2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa" gracePeriod=30 Oct 30 01:03:04.040050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.040014 5548 patch_prober.go:29] interesting pod/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:03:04.041728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.040077 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:03:04.283030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.283006 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:03:04.421567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421535 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.421759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421657 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.421759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421695 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.421759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421754 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.421977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421775 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.421977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421850 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.422093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.421994 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.422093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422025 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.422093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422086 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\") pod \"4f50c84a-2dcd-4a70-8006-cd842ce95230\" (UID: \"4f50c84a-2dcd-4a70-8006-cd842ce95230\") " Oct 30 01:03:04.422247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422122 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:04.422247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422135 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:04.422357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422268 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:04.422412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422367 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:04.422606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422579 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:04.423054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.422977 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.423054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.423002 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.423054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.423017 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f50c84a-2dcd-4a70-8006-cd842ce95230-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.423054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.423031 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.423054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.423044 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.424291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.424258 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:04.424418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.424323 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:04.424484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.424416 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "kube-api-access-jkt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:04.424709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.424692 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo" pod "4f50c84a-2dcd-4a70-8006-cd842ce95230" (UID: "4f50c84a-2dcd-4a70-8006-cd842ce95230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:04.524681 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.524650 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkt4v\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-kube-api-access-jkt4v\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.524681 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.524679 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.524881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.524691 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.524881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.524699 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f50c84a-2dcd-4a70-8006-cd842ce95230-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:04.762739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.762652 5548 generic.go:356] "Generic (PLEG): container finished" podID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerID="2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa" exitCode=0 Oct 30 01:03:04.762739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.762699 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" Oct 30 01:03:04.762739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.762715 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" event={"ID":"4f50c84a-2dcd-4a70-8006-cd842ce95230","Type":"ContainerDied","Data":"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa"} Oct 30 01:03:04.763037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.762766 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl" event={"ID":"4f50c84a-2dcd-4a70-8006-cd842ce95230","Type":"ContainerDied","Data":"ff0c286c09168ad740cf772015def2c63536f7b48e0d7b32a8072de296079708"} Oct 30 01:03:04.763037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.762792 5548 scope.go:122] "RemoveContainer" containerID="2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa" Oct 30 01:03:04.773460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.773438 5548 scope.go:122] "RemoveContainer" containerID="2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa" Oct 30 01:03:04.773739 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:04.773721 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa\": container with ID starting with 2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa not found: ID does not exist" containerID="2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa" Oct 30 01:03:04.773806 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.773755 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa"} err="failed to get container status \"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa\": rpc error: code = NotFound desc = could not find container \"2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa\": container with ID starting with 2a5cb0d4249a5a0191ea373838651041fe7d16553259a72bfc98264fcf2979aa not found: ID does not exist" Oct 30 01:03:04.817587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.817561 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:03:04.827237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:04.827215 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b61ed672-openshift-default-6cb675d767-jfnwl"] Oct 30 01:03:05.265394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:05.265358 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" path="/var/lib/kubelet/pods/4f50c84a-2dcd-4a70-8006-cd842ce95230/volumes" Oct 30 01:03:06.931820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.931789 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:06.932552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932530 5548 cpu_manager.go:414] "containerMap: removing container" podUID="dc6c7361-1991-4ba2-bed3-c463252a1b58" containerName="maas-api" Oct 30 01:03:06.932629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932557 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6c7361-1991-4ba2-bed3-c463252a1b58" containerName="maas-api" Oct 30 01:03:06.932629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932582 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" Oct 30 01:03:06.932629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932589 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" Oct 30 01:03:06.932629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932624 5548 cpu_manager.go:414] "containerMap: removing container" podUID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" containerName="maas-api" Oct 30 01:03:06.932629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932629 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" containerName="maas-api" Oct 30 01:03:06.932778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932752 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f50c84a-2dcd-4a70-8006-cd842ce95230" containerName="istio-proxy" Oct 30 01:03:06.932778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932764 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ae6b9d-ca10-4b3c-9d25-4217e5cafe10" containerName="maas-api" Oct 30 01:03:06.932840 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.932777 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6c7361-1991-4ba2-bed3-c463252a1b58" containerName="maas-api" Oct 30 01:03:06.935958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.935939 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:06.950212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.950187 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-837d7038-serving-cert\"" Oct 30 01:03:06.965905 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:06.965876 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:07.061558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.061524 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.061750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.061586 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fd6r\" (UniqueName: \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.061750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.061714 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.061893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.061848 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.163084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.163049 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.163259 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:07.163193 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-serving-cert: secret "maas-api-e2e-migrate-837d7038-serving-cert" not found Oct 30 01:03:07.163259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.163202 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.163259 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:07.163257 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls podName:9102d21a-584d-450b-aa30-70d69539ba7d nodeName:}" failed. No retries permitted until 2026-10-30 01:03:07.663239374 +0000 UTC m=+1178.505799946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls") pod "maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" (UID: "9102d21a-584d-450b-aa30-70d69539ba7d") : secret "maas-api-e2e-migrate-837d7038-serving-cert" not found Oct 30 01:03:07.163384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.163293 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fd6r\" (UniqueName: \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.163384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.163338 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.163873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.163842 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.165909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.165887 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.197938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.197457 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fd6r\" (UniqueName: \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.687727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.687689 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.690614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.690583 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") pod \"maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:07.849522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:07.849483 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:08.204324 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:08.204292 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:08.815762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:08.815640 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" event={"ID":"9102d21a-584d-450b-aa30-70d69539ba7d","Type":"ContainerStarted","Data":"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058"} Oct 30 01:03:08.816068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:08.816028 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" event={"ID":"9102d21a-584d-450b-aa30-70d69539ba7d","Type":"ContainerStarted","Data":"5bf94e8f736e5a53689c79a10f77ba7054a93a669d4178d9487ef77357da964b"} Oct 30 01:03:08.817467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:08.817447 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:08.859394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:08.859347 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" podStartSLOduration=2.554246758 podStartE2EDuration="2.859330293s" totalImagesPullingTime="305.083535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:08.855661048 +0000 UTC m=+1179.698221640" watchObservedRunningTime="2026-10-30 01:03:08.859330293 +0000 UTC m=+1179.701890918" Oct 30 01:03:13.817787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.817752 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:03:13.821460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.821440 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.834264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.834246 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:03:13.834264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.834247 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:03:13.834511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.834249 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:03:13.834511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.834436 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-txv2f\"" Oct 30 01:03:13.849723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.849700 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:03:13.928152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928123 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928171 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgznl\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928254 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928277 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928346 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928388 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:13.928515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:13.928495 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030501 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030468 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030517 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgznl\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030548 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030568 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030602 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030747 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.030916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.030838 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.031628 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.031465 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.033370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.033339 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.033370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.033353 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.033552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.033504 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.033552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.033507 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.049245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.049211 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.052838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.052813 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgznl\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xq4mm\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.133853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.133774 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:14.448559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.448359 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:03:14.448755 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:03:14.448719 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5efbe9d_b694_4bf8_9183_526e58a64f7d.slice/crio-0cc0e29a800b54fdc3fccc9c8fd538447a915a55da34e9cb5496180d77e74988 WatchSource:0}: Error finding container 0cc0e29a800b54fdc3fccc9c8fd538447a915a55da34e9cb5496180d77e74988: Status 404 returned error can't find the container with id 0cc0e29a800b54fdc3fccc9c8fd538447a915a55da34e9cb5496180d77e74988 Oct 30 01:03:14.886234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:14.886155 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" event={"ID":"b5efbe9d-b694-4bf8-9183-526e58a64f7d","Type":"ContainerStarted","Data":"0cc0e29a800b54fdc3fccc9c8fd538447a915a55da34e9cb5496180d77e74988"} Oct 30 01:03:15.855312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:15.855283 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:17.032324 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:17.032293 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:03:17.032592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:17.032365 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:03:17.913659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:17.913626 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" event={"ID":"b5efbe9d-b694-4bf8-9183-526e58a64f7d","Type":"ContainerStarted","Data":"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775"} Oct 30 01:03:17.913927 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:17.913907 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:17.972987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:17.972927 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" podStartSLOduration=2.393071647 podStartE2EDuration="4.972907161s" totalImagesPullingTime="2.579835514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:17.959520817 +0000 UTC m=+1188.802081410" watchObservedRunningTime="2026-10-30 01:03:17.972907161 +0000 UTC m=+1188.815467755" Oct 30 01:03:18.922851 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:18.922820 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:03:25.427951 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:25.427910 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:03:29.484624 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:29.484594 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:03:29.486559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:29.486535 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:03:31.052569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.052464 5548 generic.go:356] "Generic (PLEG): container finished" podID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerID="50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff" exitCode=0 Oct 30 01:03:31.052569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.052547 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerDied","Data":"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff"} Oct 30 01:03:31.053189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.053129 5548 scope.go:122] "RemoveContainer" containerID="50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff" Oct 30 01:03:31.054622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.054591 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:03:31.054701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.054662 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:03:31.054701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:31.054696 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:03:32.060822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:32.060787 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerStarted","Data":"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d"} Oct 30 01:03:32.253567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:32.253521 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:32.253787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:32.253761 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:32.253897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:32.253818 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:33.253832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:33.253800 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:33.254254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:33.253851 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:34.253614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:34.253576 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:34.253795 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:34.253644 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:35.253588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:35.253555 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:35.254068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:35.253615 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:36.253183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:36.253153 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:36.253379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:36.253216 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:37.253690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:37.253657 5548 patch_prober.go:29] interesting pod/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:03:37.254116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:37.253706 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:03:37.622555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:37.622476 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:38.258343 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:38.258314 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:38.259315 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:38.259296 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:52.642834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:52.642799 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:52.643377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:52.643035 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" podUID="9102d21a-584d-450b-aa30-70d69539ba7d" containerName="maas-api" containerID="cri-o://98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058" gracePeriod=30 Oct 30 01:03:52.970930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:52.970896 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:53.102061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.101642 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\") pod \"9102d21a-584d-450b-aa30-70d69539ba7d\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " Oct 30 01:03:53.102061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.101719 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\" (UniqueName: \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\") pod \"9102d21a-584d-450b-aa30-70d69539ba7d\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " Oct 30 01:03:53.102061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.101847 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\") pod \"9102d21a-584d-450b-aa30-70d69539ba7d\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " Oct 30 01:03:53.102061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.101992 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") pod \"9102d21a-584d-450b-aa30-70d69539ba7d\" (UID: \"9102d21a-584d-450b-aa30-70d69539ba7d\") " Oct 30 01:03:53.106137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.106101 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca" pod "9102d21a-584d-450b-aa30-70d69539ba7d" (UID: "9102d21a-584d-450b-aa30-70d69539ba7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:53.109464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.109433 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls" pod "9102d21a-584d-450b-aa30-70d69539ba7d" (UID: "9102d21a-584d-450b-aa30-70d69539ba7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:03:53.109770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.109716 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r" pod "9102d21a-584d-450b-aa30-70d69539ba7d" (UID: "9102d21a-584d-450b-aa30-70d69539ba7d"). InnerVolumeSpecName "kube-api-access-6fd6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:53.110036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.110009 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs" pod "9102d21a-584d-450b-aa30-70d69539ba7d" (UID: "9102d21a-584d-450b-aa30-70d69539ba7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:03:53.206231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.206156 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9102d21a-584d-450b-aa30-70d69539ba7d-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.206231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.206183 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fd6r\" (UniqueName: \"kubernetes.io/projected/9102d21a-584d-450b-aa30-70d69539ba7d-kube-api-access-6fd6r\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.206231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.206193 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.206231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.206202 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9102d21a-584d-450b-aa30-70d69539ba7d-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:53.246132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.246097 5548 generic.go:356] "Generic (PLEG): container finished" podID="9102d21a-584d-450b-aa30-70d69539ba7d" containerID="98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058" exitCode=0 Oct 30 01:03:53.246335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.246139 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" Oct 30 01:03:53.246335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.246189 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" event={"ID":"9102d21a-584d-450b-aa30-70d69539ba7d","Type":"ContainerDied","Data":"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058"} Oct 30 01:03:53.246335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.246281 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6" event={"ID":"9102d21a-584d-450b-aa30-70d69539ba7d","Type":"ContainerDied","Data":"5bf94e8f736e5a53689c79a10f77ba7054a93a669d4178d9487ef77357da964b"} Oct 30 01:03:53.246335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.246312 5548 scope.go:122] "RemoveContainer" containerID="98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058" Oct 30 01:03:53.257615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.257587 5548 scope.go:122] "RemoveContainer" containerID="98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058" Oct 30 01:03:53.257906 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:53.257883 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058\": container with ID starting with 98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058 not found: ID does not exist" containerID="98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058" Oct 30 01:03:53.257978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.257918 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058"} err="failed to get container status \"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058\": rpc error: code = NotFound desc = could not find container \"98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058\": container with ID starting with 98ece89f19bee76580896453cd6731a753880b321eeecc481480221d059aa058 not found: ID does not exist" Oct 30 01:03:53.293200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.293176 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:53.302299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.302275 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-837d7038-9b68867d7-tb9j6"] Oct 30 01:03:53.657259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.657207 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:03:53.659033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:53.658737 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" containerID="cri-o://b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d" gracePeriod=30 Oct 30 01:03:55.265425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:55.265383 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9102d21a-584d-450b-aa30-70d69539ba7d" path="/var/lib/kubelet/pods/9102d21a-584d-450b-aa30-70d69539ba7d/volumes" Oct 30 01:03:55.482570 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:55.482535 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:03:59.122742 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.122711 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:59.217713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217673 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.217928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217731 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.217928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217800 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.217928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217829 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.217928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217906 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.218148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217930 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.218148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.217993 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.218148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.218010 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.218148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.218040 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\") pod \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\" (UID: \"3016dc0d-8b19-4596-9ffb-01eec7d685d6\") " Oct 30 01:03:59.218703 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.218664 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:59.219615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.218953 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:59.219615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.218983 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:59.219615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.219043 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:59.219615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.219582 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:59.222806 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.222777 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "kube-api-access-62l4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:59.223224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.223185 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:59.223331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.223297 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:59.238419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.238390 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token" pod "3016dc0d-8b19-4596-9ffb-01eec7d685d6" (UID: "3016dc0d-8b19-4596-9ffb-01eec7d685d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:59.330552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330516 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330552 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330567 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330583 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62l4s\" (UniqueName: \"kubernetes.io/projected/3016dc0d-8b19-4596-9ffb-01eec7d685d6-kube-api-access-62l4s\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330596 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330609 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330622 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330637 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.330831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.330648 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3016dc0d-8b19-4596-9ffb-01eec7d685d6-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:03:59.334250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.333182 5548 generic.go:356] "Generic (PLEG): container finished" podID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerID="b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d" exitCode=0 Oct 30 01:03:59.334250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.333246 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerDied","Data":"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d"} Oct 30 01:03:59.334250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.333298 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" event={"ID":"3016dc0d-8b19-4596-9ffb-01eec7d685d6","Type":"ContainerDied","Data":"b5b8972ed387b2b56e380b6d3a52ad78d81714e651ed1e636443dc25403a355c"} Oct 30 01:03:59.334250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.333329 5548 scope.go:122] "RemoveContainer" containerID="b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d" Oct 30 01:03:59.334250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.333498 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4" Oct 30 01:03:59.349200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.349167 5548 scope.go:122] "RemoveContainer" containerID="50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff" Oct 30 01:03:59.362110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.362088 5548 scope.go:122] "RemoveContainer" containerID="b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d" Oct 30 01:03:59.362519 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:59.362493 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d\": container with ID starting with b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d not found: ID does not exist" containerID="b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d" Oct 30 01:03:59.362608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.362531 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d"} err="failed to get container status \"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d\": rpc error: code = NotFound desc = could not find container \"b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d\": container with ID starting with b7b91a1c26eff6229a1e0ff3449684ef261d883e6b633a057ed030c1db96102d not found: ID does not exist" Oct 30 01:03:59.362608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.362549 5548 scope.go:122] "RemoveContainer" containerID="50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff" Oct 30 01:03:59.362921 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:03:59.362894 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff\": container with ID starting with 50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff not found: ID does not exist" containerID="50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff" Oct 30 01:03:59.362992 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.362931 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff"} err="failed to get container status \"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff\": rpc error: code = NotFound desc = could not find container \"50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff\": container with ID starting with 50b037055be160e158997a0cf3c0778022936cfb7e2c532b2472173d9cfb56ff not found: ID does not exist" Oct 30 01:03:59.382755 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.382707 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:03:59.386175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:03:59.386148 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-837d7038-gw-openshift-default-75ffb8dbd5-jd7l4"] Oct 30 01:04:01.264765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:01.264737 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" path="/var/lib/kubelet/pods/3016dc0d-8b19-4596-9ffb-01eec7d685d6/volumes" Oct 30 01:04:08.670267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670234 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:04:08.670849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670837 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:08.670961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670851 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:08.670961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670892 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:08.670961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670897 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:08.670961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670909 5548 cpu_manager.go:414] "containerMap: removing container" podUID="9102d21a-584d-450b-aa30-70d69539ba7d" containerName="maas-api" Oct 30 01:04:08.670961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.670915 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9102d21a-584d-450b-aa30-70d69539ba7d" containerName="maas-api" Oct 30 01:04:08.671210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.671001 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:08.671210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.671010 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="9102d21a-584d-450b-aa30-70d69539ba7d" containerName="maas-api" Oct 30 01:04:08.674648 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.674627 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.686966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.686945 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-18a189-serving-cert\"" Oct 30 01:04:08.707445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.702608 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:04:08.784271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.784237 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.784447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.784379 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxht\" (UniqueName: \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.784447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.784419 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.784529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.784453 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.886646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.886589 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.889176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.889142 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxht\" (UniqueName: \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.889828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.889798 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.890010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.889896 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.890213 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:08.890193 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-serving-cert: secret "maas-api-e2e-shared-a-w2-18a189-serving-cert" not found Oct 30 01:04:08.890388 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:08.890376 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls podName:5b108e37-6075-4bb1-9204-bdf0712a9c05 nodeName:}" failed. No retries permitted until 2026-10-30 01:04:09.390351364 +0000 UTC m=+1240.232911951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls") pod "maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" (UID: "5b108e37-6075-4bb1-9204-bdf0712a9c05") : secret "maas-api-e2e-shared-a-w2-18a189-serving-cert" not found Oct 30 01:04:08.891165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.891139 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.894122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.894097 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:08.910757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:08.910724 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxht\" (UniqueName: \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:09.396567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:09.396533 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:09.399192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:09.399160 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:09.587938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:09.587906 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:09.905917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:09.905883 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:04:10.425363 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:10.425325 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" event={"ID":"5b108e37-6075-4bb1-9204-bdf0712a9c05","Type":"ContainerStarted","Data":"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa"} Oct 30 01:04:10.425534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:10.425387 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:10.425534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:10.425407 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" event={"ID":"5b108e37-6075-4bb1-9204-bdf0712a9c05","Type":"ContainerStarted","Data":"934e1b0e6cff57482ad4ba2dba43a1f997cd8d95534254b53f3ea246aaf42a5b"} Oct 30 01:04:10.461461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:10.461365 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" podStartSLOduration=2.169030016 podStartE2EDuration="2.461351074s" totalImagesPullingTime="292.321058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:10.458691999 +0000 UTC m=+1241.301252591" watchObservedRunningTime="2026-10-30 01:04:10.461351074 +0000 UTC m=+1241.303911666" Oct 30 01:04:16.437148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:16.437115 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:04:21.497936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.497893 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:04:21.498816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.498797 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3016dc0d-8b19-4596-9ffb-01eec7d685d6" containerName="istio-proxy" Oct 30 01:04:21.501675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.501614 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.525677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.525645 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:04:21.610285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610250 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610294 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610452 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610494 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610617 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610699 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610648 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610699 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610672 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610728 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjzw\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.610918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.610903 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.711993 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.711963 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712001 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712081 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712102 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712117 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712136 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjzw\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712169 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712362 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712411 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712455 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712546 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712604 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712778 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.712874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.712788 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.714828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.714803 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.714954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.714883 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.729077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.729052 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.729712 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.729689 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjzw\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:21.827263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:21.827170 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:22.152744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.152715 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:04:22.153362 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:04:22.153338 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6672ba_a11c_4a3e_b9a2_3b9562e6cce8.slice/crio-d0bf0c49007985a6bbd4831aaf0c898f1ba8ae7c1d69a82971a203d94eecc231 WatchSource:0}: Error finding container d0bf0c49007985a6bbd4831aaf0c898f1ba8ae7c1d69a82971a203d94eecc231: Status 404 returned error can't find the container with id d0bf0c49007985a6bbd4831aaf0c898f1ba8ae7c1d69a82971a203d94eecc231 Oct 30 01:04:22.155932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.155895 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.156025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.155965 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.156025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.156018 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.237081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.237053 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:04:22.240339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.240319 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.268354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.268320 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328472 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328592 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjht\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328658 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328763 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328847 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328918 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.328974 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.329106 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.329408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.329157 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430288 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430439 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430528 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430579 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430702 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430854 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430832 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.430957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430937 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430971 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.430991 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.431044 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.431089 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjht\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.431180 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.431244 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.431725 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.431704 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.433385 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.433342 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.433925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.433901 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.448650 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.448621 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.449274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.449249 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjht\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.563089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.563038 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:22.570134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.566732 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" event={"ID":"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8","Type":"ContainerStarted","Data":"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b"} Oct 30 01:04:22.570134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.566803 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" event={"ID":"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8","Type":"ContainerStarted","Data":"d0bf0c49007985a6bbd4831aaf0c898f1ba8ae7c1d69a82971a203d94eecc231"} Oct 30 01:04:22.608340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.608279 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podStartSLOduration=1.608257337 podStartE2EDuration="1.608257337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:22.605757509 +0000 UTC m=+1253.448318122" watchObservedRunningTime="2026-10-30 01:04:22.608257337 +0000 UTC m=+1253.450817931" Oct 30 01:04:22.827941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.827833 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:22.837496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.837472 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:04:22.841637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.841617 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:22.856995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.856969 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-eacea3-serving-cert\"" Oct 30 01:04:22.860895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.860847 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:04:22.907258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.907231 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:04:22.912015 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:04:22.911986 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda9ec9_f322_44fe_b855_83f52ed8f260.slice/crio-87d1013eaa0b89738eb5db05bf3e74aa9657935fc817133efd0973d89b38460c WatchSource:0}: Error finding container 87d1013eaa0b89738eb5db05bf3e74aa9657935fc817133efd0973d89b38460c: Status 404 returned error can't find the container with id 87d1013eaa0b89738eb5db05bf3e74aa9657935fc817133efd0973d89b38460c Oct 30 01:04:22.915598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.915553 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.915716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.915639 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.915716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.915683 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:22.950316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.950280 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:22.950424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.950389 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:22.950474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.950458 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflh5\" (UniqueName: \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:22.950746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:22.950719 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.052225 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.052337 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.052404 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.052486 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflh5\" (UniqueName: \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.053271 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.053442 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:23.053420 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-serving-cert: secret "maas-api-e2e-shared-a-w0-eacea3-serving-cert" not found Oct 30 01:04:23.054125 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:23.053479 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls podName:8d68a313-eb7f-4500-9f27-ec1337fcdb7b nodeName:}" failed. No retries permitted until 2026-10-30 01:04:23.553460539 +0000 UTC m=+1254.396021110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls") pod "maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" (UID: "8d68a313-eb7f-4500-9f27-ec1337fcdb7b") : secret "maas-api-e2e-shared-a-w0-eacea3-serving-cert" not found Oct 30 01:04:23.057378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.057354 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.112365 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.112261 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflh5\" (UniqueName: \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.561209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.561176 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.563960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.563937 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.613314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.613269 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" event={"ID":"9dda9ec9-f322-44fe-b855-83f52ed8f260","Type":"ContainerStarted","Data":"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c"} Oct 30 01:04:23.613502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.613445 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" event={"ID":"9dda9ec9-f322-44fe-b855-83f52ed8f260","Type":"ContainerStarted","Data":"87d1013eaa0b89738eb5db05bf3e74aa9657935fc817133efd0973d89b38460c"} Oct 30 01:04:23.653875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.653802 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podStartSLOduration=1.653784065 podStartE2EDuration="1.653784065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.650973279 +0000 UTC m=+1254.493533874" watchObservedRunningTime="2026-10-30 01:04:23.653784065 +0000 UTC m=+1254.496344715" Oct 30 01:04:23.757205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.757139 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:23.836943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.836729 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:23.836943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:23.836806 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:24.354685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.354396 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:04:24.564642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.564609 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:24.566112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.566088 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:24.566235 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.566134 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:24.635508 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.635383 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" event={"ID":"8d68a313-eb7f-4500-9f27-ec1337fcdb7b","Type":"ContainerStarted","Data":"6a814c3cda2f11929970b3209733092ccb0c154eda8e84536ce0e69b3a251bef"} Oct 30 01:04:24.827486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.827459 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:24.827625 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:24.827500 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:25.438512 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:25.438471 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:04:25.564136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.564102 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:25.564328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.564157 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:25.645035 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.644997 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" event={"ID":"8d68a313-eb7f-4500-9f27-ec1337fcdb7b","Type":"ContainerStarted","Data":"0dc12db12f738f50e99e0fe403bcb86718265a631e054e5549e851435ebf1e7b"} Oct 30 01:04:25.714938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.714360 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" podStartSLOduration=3.333587224 podStartE2EDuration="3.714340294s" totalImagesPullingTime="380.75307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:25.712563982 +0000 UTC m=+1256.555124576" watchObservedRunningTime="2026-10-30 01:04:25.714340294 +0000 UTC m=+1256.556900888" Oct 30 01:04:25.832597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.832560 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:25.832769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:25.832626 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:26.564050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:26.564016 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:26.564247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:26.564065 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:26.652257 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:26.652223 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:26.828590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:26.828498 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:26.828590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:26.828553 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:27.564682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:27.564650 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:27.564887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:27.564697 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:27.827841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:27.827762 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:27.827841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:27.827818 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:28.564373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:28.564337 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:28.564541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:28.564405 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:28.828403 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:28.828318 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:28.828403 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:28.828376 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 01:04:29.564226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:29.564194 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:04:29.564424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:29.564251 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 01:04:29.833069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:29.832992 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:29.833683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:29.833301 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:29.834223 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:29.834204 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:04:30.568930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:30.568895 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:30.569301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:30.569282 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:30.570029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:30.570009 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:04:32.667693 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:32.667666 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:04:55.434387 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:04:55.434277 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:04:56.288261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.288225 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:04:56.301415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.301332 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.320092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.320063 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:04:56.320457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.320438 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-17d855-openshift-default-dockercfg-k5qv4\"" Oct 30 01:04:56.414443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.414383 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.414819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.414782 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415035 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415011 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415160 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415304 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415339 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415391 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415419 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.415511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.415443 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmg4\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.529310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.529269 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.530479 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.530456 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.531232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.531212 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.531727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.531707 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.531924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.531907 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.535957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.535936 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmg4\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.536575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.536553 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.537469 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.537452 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.537887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.537380 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.538001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.531627 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.538091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.530944 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.538167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.537804 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.538401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.538356 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.538835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.538723 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.555747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.555716 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.556660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.556112 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.565978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.564795 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmg4\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.574247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.573152 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\") pod \"e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.629268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.625181 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:56.985359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.985311 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:04:56.988234 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:04:56.988207 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd915a47_3112_46bd_ba50_1bd19a846192.slice/crio-d550c85439f48c8229d7f0b23b1fc03906e5dc198602bb45f74a96df892dbc5c WatchSource:0}: Error finding container d550c85439f48c8229d7f0b23b1fc03906e5dc198602bb45f74a96df892dbc5c: Status 404 returned error can't find the container with id d550c85439f48c8229d7f0b23b1fc03906e5dc198602bb45f74a96df892dbc5c Oct 30 01:04:56.991439 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.991391 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:56.991556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.991481 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:56.991556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:56.991529 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:04:57.053156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:57.053117 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerStarted","Data":"d550c85439f48c8229d7f0b23b1fc03906e5dc198602bb45f74a96df892dbc5c"} Oct 30 01:04:58.080414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:58.080328 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerStarted","Data":"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad"} Oct 30 01:04:58.629072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:58.629039 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:58.649364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:58.648760 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:58.694879 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:58.691979 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podStartSLOduration=2.6919588660000002 podStartE2EDuration="2.691958866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:58.133973661 +0000 UTC m=+1288.976534255" watchObservedRunningTime="2026-10-30 01:04:58.691958866 +0000 UTC m=+1289.534519461" Oct 30 01:04:59.087557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:59.087531 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:04:59.088625 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:04:59.088603 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:05:24.532137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.532100 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:05:24.536040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.536011 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.568276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.561556 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-69dc90-serving-cert\"" Oct 30 01:05:24.568276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.566387 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.568276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.566671 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9rj\" (UniqueName: \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.568276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.566702 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.568276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.566757 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.568672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.568513 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:05:24.668582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.668549 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9rj\" (UniqueName: \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.668582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.668587 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.668783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.668655 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.668825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.668778 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.669423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.669400 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.671183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.671159 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.671339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.671320 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.687853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.687829 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9rj\" (UniqueName: \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\") pod \"maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:24.850789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:24.850713 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:25.179671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:25.179635 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:05:25.181919 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:05:25.181888 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00325850_ceb0_418c_b0e6_f0b2332711b6.slice/crio-f16090bf53c39f74c19d33b5997b6aed5843156351d477ed44632f0b6bd80bb3 WatchSource:0}: Error finding container f16090bf53c39f74c19d33b5997b6aed5843156351d477ed44632f0b6bd80bb3: Status 404 returned error can't find the container with id f16090bf53c39f74c19d33b5997b6aed5843156351d477ed44632f0b6bd80bb3 Oct 30 01:05:25.281131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:25.281091 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" event={"ID":"00325850-ceb0-418c-b0e6-f0b2332711b6","Type":"ContainerStarted","Data":"f16090bf53c39f74c19d33b5997b6aed5843156351d477ed44632f0b6bd80bb3"} Oct 30 01:05:25.432766 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:05:25.432664 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:05:26.289345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:26.289310 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" event={"ID":"00325850-ceb0-418c-b0e6-f0b2332711b6","Type":"ContainerStarted","Data":"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4"} Oct 30 01:05:26.289816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:26.289396 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:26.328064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:26.328019 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" podStartSLOduration=2.014538723 podStartE2EDuration="2.328005265s" totalImagesPullingTime="313.466542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:26.323806069 +0000 UTC m=+1317.166366661" watchObservedRunningTime="2026-10-30 01:05:26.328005265 +0000 UTC m=+1317.170565858" Oct 30 01:05:32.301833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:32.301804 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:05:50.217985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.217954 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8768-2272c9"] Oct 30 01:05:50.222234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.222213 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:50.238530 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.238504 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-2272c9"] Oct 30 01:05:50.241544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.241518 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:05:50.251452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.251417 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncfx\" (UniqueName: \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\") pod \"test-curl-8768-2272c9\" (UID: \"6a47394d-9c02-4b99-b1af-ca9e2b1c0456\") " pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:50.358817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.356674 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncfx\" (UniqueName: \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\") pod \"test-curl-8768-2272c9\" (UID: \"6a47394d-9c02-4b99-b1af-ca9e2b1c0456\") " pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:50.416313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.412721 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncfx\" (UniqueName: \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\") pod \"test-curl-8768-2272c9\" (UID: \"6a47394d-9c02-4b99-b1af-ca9e2b1c0456\") " pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:50.541110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.541083 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:50.853391 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:50.853359 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-2272c9"] Oct 30 01:05:50.854021 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:05:50.853993 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a47394d_9c02_4b99_b1af_ca9e2b1c0456.slice/crio-56b71f938fb5fe17960976e1fbee4a6fb496c8e54ef6671ccf4bb8fe650c860a WatchSource:0}: Error finding container 56b71f938fb5fe17960976e1fbee4a6fb496c8e54ef6671ccf4bb8fe650c860a: Status 404 returned error can't find the container with id 56b71f938fb5fe17960976e1fbee4a6fb496c8e54ef6671ccf4bb8fe650c860a Oct 30 01:05:51.501895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.501822 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-2272c9" event={"ID":"6a47394d-9c02-4b99-b1af-ca9e2b1c0456","Type":"ContainerStarted","Data":"54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c"} Oct 30 01:05:51.502350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.501906 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-2272c9" event={"ID":"6a47394d-9c02-4b99-b1af-ca9e2b1c0456","Type":"ContainerStarted","Data":"56b71f938fb5fe17960976e1fbee4a6fb496c8e54ef6671ccf4bb8fe650c860a"} Oct 30 01:05:51.503497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.503472 5548 generic.go:356] "Generic (PLEG): container finished" podID="bd915a47-3112-46bd-ba50-1bd19a846192" containerID="5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad" exitCode=0 Oct 30 01:05:51.503608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.503540 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerDied","Data":"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad"} Oct 30 01:05:51.503917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.503905 5548 scope.go:122] "RemoveContainer" containerID="5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad" Oct 30 01:05:51.505587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.505554 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:51.505689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.505640 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:51.505747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.505682 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:51.537189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:51.537143 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8768-2272c9" podStartSLOduration=1.537128929 podStartE2EDuration="1.537128929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:51.532427049 +0000 UTC m=+1342.374987663" watchObservedRunningTime="2026-10-30 01:05:51.537128929 +0000 UTC m=+1342.379689522" Oct 30 01:05:52.425152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.425049 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8768-2272c9"] Oct 30 01:05:52.431344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.430220 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8768-2272c9"] Oct 30 01:05:52.545879 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.543631 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerStarted","Data":"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17"} Oct 30 01:05:52.626430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.626396 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:05:52.626642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.626502 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:52.626642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:52.626543 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:53.327611 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.327574 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:05:53.328393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.328364 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" containerName="test-curl-8768-2272c9" Oct 30 01:05:53.328393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.328391 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" containerName="test-curl-8768-2272c9" Oct 30 01:05:53.328575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.328546 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" containerName="test-curl-8768-2272c9" Oct 30 01:05:53.331607 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.331586 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.408824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.408785 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:05:53.440555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.440517 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-288da6-openshift-default-dockercfg-bbq4z\"" Oct 30 01:05:53.466364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466322 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.466546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466485 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvb2\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.466607 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466564 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.466701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466680 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.466824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466810 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.466946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466846 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.467007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.466966 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.471907 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.467071 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.471907 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.467149 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.550948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.550898 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8768-2272c9" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" containerName="test-curl-8768-2272c9" containerID="cri-o://54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c" gracePeriod=2 Oct 30 01:05:53.576975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.576917 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvb2\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.577677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.577615 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.578735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.578627 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.593415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.593292 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.593839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.593817 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.594560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.594524 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.595116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.595097 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.595848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.595829 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.599549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.596031 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.622440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.622407 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.623396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.594982 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.623617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.600438 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.623771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.622711 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.624055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.622929 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.624171 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.594307 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.629952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.629387 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:53.629952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.629456 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:53.630674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.630621 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.643421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.643328 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvb2\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.648007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.647591 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\") pod \"e2e-api-b-288da6-openshift-default-68599494b9-5kwfj\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:53.648007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:53.647823 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:54.038588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.038557 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:05:54.038795 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:05:54.038776 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a03d5e_3270_4c4a_946e_54a9a8a348eb.slice/crio-554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8 WatchSource:0}: Error finding container 554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8: Status 404 returned error can't find the container with id 554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8 Oct 30 01:05:54.041508 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.041468 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:54.041614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.041555 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:54.041614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.041598 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:05:54.559815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.559773 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerStarted","Data":"67d5dcdb1962a743d7aa7f2225c50135a4528b37a1a78dfa12379e9050a4b317"} Oct 30 01:05:54.560309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.559872 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerStarted","Data":"554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8"} Oct 30 01:05:54.616269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.616200 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podStartSLOduration=1.616184249 podStartE2EDuration="1.616184249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:54.609506047 +0000 UTC m=+1345.452066676" watchObservedRunningTime="2026-10-30 01:05:54.616184249 +0000 UTC m=+1345.458744842" Oct 30 01:05:54.628272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.628247 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:54.628424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.628392 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:54.649067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:54.649042 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:55.428127 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:05:55.428091 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:05:55.626167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.626135 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:55.626623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.626205 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:55.654328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.654306 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:55.810056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.810032 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:55.863848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.863810 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\" (UniqueName: \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\") pod \"6a47394d-9c02-4b99-b1af-ca9e2b1c0456\" (UID: \"6a47394d-9c02-4b99-b1af-ca9e2b1c0456\") " Oct 30 01:05:55.866044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.866017 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx" pod "6a47394d-9c02-4b99-b1af-ca9e2b1c0456" (UID: "6a47394d-9c02-4b99-b1af-ca9e2b1c0456"). InnerVolumeSpecName "kube-api-access-4ncfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:55.966022 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:55.965945 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ncfx\" (UniqueName: \"kubernetes.io/projected/6a47394d-9c02-4b99-b1af-ca9e2b1c0456-kube-api-access-4ncfx\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:05:56.603467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.603436 5548 generic.go:356] "Generic (PLEG): container finished" podID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" containerID="54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c" exitCode=137 Oct 30 01:05:56.603663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.603478 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:56.603663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.603531 5548 scope.go:122] "RemoveContainer" containerID="54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c" Oct 30 01:05:56.604120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.604095 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:56.605094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.605078 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:05:56.615060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.614164 5548 scope.go:122] "RemoveContainer" containerID="54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c" Oct 30 01:05:56.615060 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:05:56.614450 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c\": container with ID starting with 54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c not found: ID does not exist" containerID="54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c" Oct 30 01:05:56.615060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.614478 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c"} err="failed to get container status \"54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c\": rpc error: code = NotFound desc = could not find container \"54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c\": container with ID starting with 54563767353a2ad61217cf25d0a19376617bcfc65aaa5442785e6881bf71b56c not found: ID does not exist" Oct 30 01:05:56.626413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.626391 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:56.626688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:56.626432 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:56.661749 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:05:56.661708 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-2272c9\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" pod="opendatahub/test-curl-8768-2272c9" Oct 30 01:05:57.271891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.269160 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a47394d-9c02-4b99-b1af-ca9e2b1c0456" path="/var/lib/kubelet/pods/6a47394d-9c02-4b99-b1af-ca9e2b1c0456/volumes" Oct 30 01:05:57.625924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.625809 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:05:57.625924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.625894 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:05:57.698419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.698384 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8753-b4308f"] Oct 30 01:05:57.704837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.704816 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:05:57.717842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.717814 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-b4308f"] Oct 30 01:05:57.738319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.738288 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:05:57.827222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.827189 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62nq\" (UniqueName: \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\") pod \"test-curl-8753-b4308f\" (UID: \"28c12e93-e743-4e51-8fb6-685401488f51\") " pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:05:57.928102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.928069 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62nq\" (UniqueName: \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\") pod \"test-curl-8753-b4308f\" (UID: \"28c12e93-e743-4e51-8fb6-685401488f51\") " pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:05:57.948262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:57.948220 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62nq\" (UniqueName: \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\") pod \"test-curl-8753-b4308f\" (UID: \"28c12e93-e743-4e51-8fb6-685401488f51\") " pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:05:58.021192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.021157 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:05:58.318189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.318163 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-b4308f"] Oct 30 01:05:58.319700 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:05:58.319663 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28c12e93_e743_4e51_8fb6_685401488f51.slice/crio-297b65ad0c1a18fb45337f0f51270579c310c59fbbc9cdfb6a02ed73f51121fd WatchSource:0}: Error finding container 297b65ad0c1a18fb45337f0f51270579c310c59fbbc9cdfb6a02ed73f51121fd: Status 404 returned error can't find the container with id 297b65ad0c1a18fb45337f0f51270579c310c59fbbc9cdfb6a02ed73f51121fd Oct 30 01:05:58.621591 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.621495 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-b4308f" event={"ID":"28c12e93-e743-4e51-8fb6-685401488f51","Type":"ContainerStarted","Data":"33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0"} Oct 30 01:05:58.621591 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.621546 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-b4308f" event={"ID":"28c12e93-e743-4e51-8fb6-685401488f51","Type":"ContainerStarted","Data":"297b65ad0c1a18fb45337f0f51270579c310c59fbbc9cdfb6a02ed73f51121fd"} Oct 30 01:05:58.630393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.630368 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:05:58.630751 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.630729 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:05:58.631398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.631377 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:05:58.655005 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:58.654954 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8753-b4308f" podStartSLOduration=1.654938687 podStartE2EDuration="1.654938687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.653297034 +0000 UTC m=+1349.495857627" watchObservedRunningTime="2026-10-30 01:05:58.654938687 +0000 UTC m=+1349.497499281" Oct 30 01:05:59.457699 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.457663 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8753-b4308f"] Oct 30 01:05:59.463762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.463729 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8753-b4308f"] Oct 30 01:05:59.605731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.605701 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8753-c6bc00"] Oct 30 01:05:59.606375 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.606361 5548 cpu_manager.go:414] "containerMap: removing container" podUID="28c12e93-e743-4e51-8fb6-685401488f51" containerName="test-curl-8753-b4308f" Oct 30 01:05:59.606460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.606376 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c12e93-e743-4e51-8fb6-685401488f51" containerName="test-curl-8753-b4308f" Oct 30 01:05:59.606499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.606477 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c12e93-e743-4e51-8fb6-685401488f51" containerName="test-curl-8753-b4308f" Oct 30 01:05:59.609918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.609894 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:05:59.626167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.626143 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-c6bc00"] Oct 30 01:05:59.712412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.712321 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9xz\" (UniqueName: \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\") pod \"test-curl-8753-c6bc00\" (UID: \"b9af457b-8ba2-4c37-ad0e-8d6012970f10\") " pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:05:59.814457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.814424 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9xz\" (UniqueName: \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\") pod \"test-curl-8753-c6bc00\" (UID: \"b9af457b-8ba2-4c37-ad0e-8d6012970f10\") " pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:05:59.831053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.831028 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9xz\" (UniqueName: \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\") pod \"test-curl-8753-c6bc00\" (UID: \"b9af457b-8ba2-4c37-ad0e-8d6012970f10\") " pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:05:59.923351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:05:59.923318 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:00.226221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:00.225947 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-c6bc00"] Oct 30 01:06:00.228756 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:06:00.227341 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9af457b_8ba2_4c37_ad0e_8d6012970f10.slice/crio-7a6a5f52392e4a6eeefca55e59da4b393b574df46e6cdc768d9fdb78b632db8f WatchSource:0}: Error finding container 7a6a5f52392e4a6eeefca55e59da4b393b574df46e6cdc768d9fdb78b632db8f: Status 404 returned error can't find the container with id 7a6a5f52392e4a6eeefca55e59da4b393b574df46e6cdc768d9fdb78b632db8f Oct 30 01:06:00.641088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:00.640993 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-c6bc00" event={"ID":"b9af457b-8ba2-4c37-ad0e-8d6012970f10","Type":"ContainerStarted","Data":"e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c"} Oct 30 01:06:00.641088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:00.641041 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-c6bc00" event={"ID":"b9af457b-8ba2-4c37-ad0e-8d6012970f10","Type":"ContainerStarted","Data":"7a6a5f52392e4a6eeefca55e59da4b393b574df46e6cdc768d9fdb78b632db8f"} Oct 30 01:06:00.641627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:00.641086 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8753-b4308f" podUID="28c12e93-e743-4e51-8fb6-685401488f51" containerName="test-curl-8753-b4308f" containerID="cri-o://33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0" gracePeriod=2 Oct 30 01:06:00.653277 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:00.653255 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-b4308f\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="28c12e93-e743-4e51-8fb6-685401488f51" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:06:00.705935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:00.701213 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8753-c6bc00" podStartSLOduration=1.701194589 podStartE2EDuration="1.701194589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:00.698395549 +0000 UTC m=+1351.540956143" watchObservedRunningTime="2026-10-30 01:06:00.701194589 +0000 UTC m=+1351.543755184" Oct 30 01:06:00.710519 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:00.710486 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-b4308f\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="28c12e93-e743-4e51-8fb6-685401488f51" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:06:01.569447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.569415 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8753-c6bc00"] Oct 30 01:06:01.581301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.581264 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8753-c6bc00"] Oct 30 01:06:01.721185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.721155 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8753-9eac93"] Oct 30 01:06:01.722021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.722000 5548 cpu_manager.go:414] "containerMap: removing container" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" containerName="test-curl-8753-c6bc00" Oct 30 01:06:01.722021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.722021 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" containerName="test-curl-8753-c6bc00" Oct 30 01:06:01.722201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.722187 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" containerName="test-curl-8753-c6bc00" Oct 30 01:06:01.724744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.724725 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:01.741734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.741704 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-9eac93"] Oct 30 01:06:01.745419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.745396 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkfl\" (UniqueName: \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\") pod \"test-curl-8753-9eac93\" (UID: \"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b\") " pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:01.846964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.846844 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkfl\" (UniqueName: \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\") pod \"test-curl-8753-9eac93\" (UID: \"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b\") " pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:01.863513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:01.863483 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkfl\" (UniqueName: \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\") pod \"test-curl-8753-9eac93\" (UID: \"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b\") " pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:02.037054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.037025 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:02.338451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.338427 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-9eac93"] Oct 30 01:06:02.340735 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:06:02.340703 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f17a6b4_3be3_4d82_a1f0_17fc663a4a1b.slice/crio-e366e4063a38516c209545bb2c745e0ce3a08842816bf53d2e09d608ddc186ad WatchSource:0}: Error finding container e366e4063a38516c209545bb2c745e0ce3a08842816bf53d2e09d608ddc186ad: Status 404 returned error can't find the container with id e366e4063a38516c209545bb2c745e0ce3a08842816bf53d2e09d608ddc186ad Oct 30 01:06:02.656791 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.656760 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-9eac93" event={"ID":"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b","Type":"ContainerStarted","Data":"05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce"} Oct 30 01:06:02.657142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.656803 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-9eac93" event={"ID":"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b","Type":"ContainerStarted","Data":"e366e4063a38516c209545bb2c745e0ce3a08842816bf53d2e09d608ddc186ad"} Oct 30 01:06:02.657142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.656891 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8753-c6bc00" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" containerName="test-curl-8753-c6bc00" containerID="cri-o://e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c" gracePeriod=2 Oct 30 01:06:02.669092 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:02.669066 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-c6bc00\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:02.705300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.705243 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8753-9eac93" podStartSLOduration=1.705226981 podStartE2EDuration="1.705226981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:02.700602808 +0000 UTC m=+1353.543163402" watchObservedRunningTime="2026-10-30 01:06:02.705226981 +0000 UTC m=+1353.547787589" Oct 30 01:06:02.711520 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:02.711491 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-c6bc00\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:02.800926 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.800900 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:06:02.812147 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:02.812109 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-c6bc00\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:02.862253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.862224 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\" (UniqueName: \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\") pod \"28c12e93-e743-4e51-8fb6-685401488f51\" (UID: \"28c12e93-e743-4e51-8fb6-685401488f51\") " Oct 30 01:06:02.864498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.864469 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq" pod "28c12e93-e743-4e51-8fb6-685401488f51" (UID: "28c12e93-e743-4e51-8fb6-685401488f51"). InnerVolumeSpecName "kube-api-access-j62nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:02.964281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:02.964197 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j62nq\" (UniqueName: \"kubernetes.io/projected/28c12e93-e743-4e51-8fb6-685401488f51-kube-api-access-j62nq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:06:03.264971 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.264882 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c12e93-e743-4e51-8fb6-685401488f51" path="/var/lib/kubelet/pods/28c12e93-e743-4e51-8fb6-685401488f51/volumes" Oct 30 01:06:03.489135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.489102 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8753-9eac93"] Oct 30 01:06:03.497735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.497481 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8753-9eac93"] Oct 30 01:06:03.636147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.636065 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8753-bc628c"] Oct 30 01:06:03.636899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.636840 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" containerName="test-curl-8753-9eac93" Oct 30 01:06:03.636899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.636878 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" containerName="test-curl-8753-9eac93" Oct 30 01:06:03.637089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.637049 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" containerName="test-curl-8753-9eac93" Oct 30 01:06:03.640060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.640041 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:03.672252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.672101 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-bc628c"] Oct 30 01:06:03.674484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.674456 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flk7\" (UniqueName: \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\") pod \"test-curl-8753-bc628c\" (UID: \"937292f6-aede-414c-a73d-a05ae812e483\") " pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:03.689899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.688177 5548 generic.go:356] "Generic (PLEG): container finished" podID="28c12e93-e743-4e51-8fb6-685401488f51" containerID="33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0" exitCode=137 Oct 30 01:06:03.689899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.689077 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-b4308f" Oct 30 01:06:03.689899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.689327 5548 scope.go:122] "RemoveContainer" containerID="33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0" Oct 30 01:06:03.743211 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.743181 5548 scope.go:122] "RemoveContainer" containerID="33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0" Oct 30 01:06:03.744068 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:03.744039 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0\": container with ID starting with 33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0 not found: ID does not exist" containerID="33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0" Oct 30 01:06:03.744208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.744082 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0"} err="failed to get container status \"33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0\": rpc error: code = NotFound desc = could not find container \"33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0\": container with ID starting with 33c66b3fc0c7c33df6bab6176ca9b3688c0efaaca843f062901d58fa033bf6b0 not found: ID does not exist" Oct 30 01:06:03.775525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.775499 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flk7\" (UniqueName: \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\") pod \"test-curl-8753-bc628c\" (UID: \"937292f6-aede-414c-a73d-a05ae812e483\") " pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:03.794414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.794382 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flk7\" (UniqueName: \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\") pod \"test-curl-8753-bc628c\" (UID: \"937292f6-aede-414c-a73d-a05ae812e483\") " pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:03.970969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:03.970939 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:04.275566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.275478 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-bc628c"] Oct 30 01:06:04.281364 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:06:04.281326 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937292f6_aede_414c_a73d_a05ae812e483.slice/crio-bb949a0ef9f1d98eee384e2ca70d46b81fb7215d803ae2f3c57cf06f70b599cf WatchSource:0}: Error finding container bb949a0ef9f1d98eee384e2ca70d46b81fb7215d803ae2f3c57cf06f70b599cf: Status 404 returned error can't find the container with id bb949a0ef9f1d98eee384e2ca70d46b81fb7215d803ae2f3c57cf06f70b599cf Oct 30 01:06:04.698433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.698403 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-bc628c" event={"ID":"937292f6-aede-414c-a73d-a05ae812e483","Type":"ContainerStarted","Data":"53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30"} Oct 30 01:06:04.698578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.698449 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-bc628c" event={"ID":"937292f6-aede-414c-a73d-a05ae812e483","Type":"ContainerStarted","Data":"bb949a0ef9f1d98eee384e2ca70d46b81fb7215d803ae2f3c57cf06f70b599cf"} Oct 30 01:06:04.699873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.699833 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8753-9eac93" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" containerName="test-curl-8753-9eac93" containerID="cri-o://05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce" gracePeriod=2 Oct 30 01:06:04.740258 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:04.740211 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-9eac93\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:04.740708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.740656 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8753-bc628c" podStartSLOduration=1.7406313949999999 podStartE2EDuration="1.740631395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:04.728965803 +0000 UTC m=+1355.571526397" watchObservedRunningTime="2026-10-30 01:06:04.740631395 +0000 UTC m=+1355.583191988" Oct 30 01:06:04.870244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.870220 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:04.881346 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:04.881317 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-9eac93\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:04.996833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.996805 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\" (UniqueName: \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\") pod \"b9af457b-8ba2-4c37-ad0e-8d6012970f10\" (UID: \"b9af457b-8ba2-4c37-ad0e-8d6012970f10\") " Oct 30 01:06:04.999764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:04.999719 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz" pod "b9af457b-8ba2-4c37-ad0e-8d6012970f10" (UID: "b9af457b-8ba2-4c37-ad0e-8d6012970f10"). InnerVolumeSpecName "kube-api-access-qj9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:05.099179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.099150 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9xz\" (UniqueName: \"kubernetes.io/projected/b9af457b-8ba2-4c37-ad0e-8d6012970f10-kube-api-access-qj9xz\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:06:05.266947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.266834 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" path="/var/lib/kubelet/pods/b9af457b-8ba2-4c37-ad0e-8d6012970f10/volumes" Oct 30 01:06:05.547374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.547286 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8753-bc628c"] Oct 30 01:06:05.553203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.553173 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8753-bc628c"] Oct 30 01:06:05.708424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.708393 5548 generic.go:356] "Generic (PLEG): container finished" podID="b9af457b-8ba2-4c37-ad0e-8d6012970f10" containerID="e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c" exitCode=137 Oct 30 01:06:05.708631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.708435 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-c6bc00" Oct 30 01:06:05.708631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.708490 5548 scope.go:122] "RemoveContainer" containerID="e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c" Oct 30 01:06:05.719089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.718894 5548 scope.go:122] "RemoveContainer" containerID="e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c" Oct 30 01:06:05.719233 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:05.719203 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c\": container with ID starting with e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c not found: ID does not exist" containerID="e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c" Oct 30 01:06:05.719280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:05.719242 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c"} err="failed to get container status \"e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c\": rpc error: code = NotFound desc = could not find container \"e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c\": container with ID starting with e546733909f007120e4f4b0a2a5079d5142fe534f326ec810c4b256ae732f14c not found: ID does not exist" Oct 30 01:06:06.717642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:06.717517 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8753-bc628c" podUID="937292f6-aede-414c-a73d-a05ae812e483" containerName="test-curl-8753-bc628c" containerID="cri-o://53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30" gracePeriod=2 Oct 30 01:06:06.729054 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:06.729023 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-bc628c\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="937292f6-aede-414c-a73d-a05ae812e483" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:06.889818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:06.889796 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:06.901085 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:06.901059 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-bc628c\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="937292f6-aede-414c-a73d-a05ae812e483" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:07.031727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.031642 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\" (UniqueName: \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\") pod \"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b\" (UID: \"5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b\") " Oct 30 01:06:07.034083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.034053 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl" pod "5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" (UID: "5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b"). InnerVolumeSpecName "kube-api-access-whkfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:07.133524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.133490 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whkfl\" (UniqueName: \"kubernetes.io/projected/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b-kube-api-access-whkfl\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:06:07.265412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.265384 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" path="/var/lib/kubelet/pods/5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b/volumes" Oct 30 01:06:07.726935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.726903 5548 generic.go:356] "Generic (PLEG): container finished" podID="5f17a6b4-3be3-4d82-a1f0-17fc663a4a1b" containerID="05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce" exitCode=137 Oct 30 01:06:07.727407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.726949 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-9eac93" Oct 30 01:06:07.727407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.727010 5548 scope.go:122] "RemoveContainer" containerID="05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce" Oct 30 01:06:07.737666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.737647 5548 scope.go:122] "RemoveContainer" containerID="05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce" Oct 30 01:06:07.737965 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:07.737936 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce\": container with ID starting with 05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce not found: ID does not exist" containerID="05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce" Oct 30 01:06:07.738039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:07.737978 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce"} err="failed to get container status \"05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce\": rpc error: code = NotFound desc = could not find container \"05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce\": container with ID starting with 05e9658d23400d26d083e718c9c2f984dfa31dec3b452ed25c8eaf4c27d160ce not found: ID does not exist" Oct 30 01:06:08.969998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:08.969971 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:09.061462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.061374 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\" (UniqueName: \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\") pod \"937292f6-aede-414c-a73d-a05ae812e483\" (UID: \"937292f6-aede-414c-a73d-a05ae812e483\") " Oct 30 01:06:09.063700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.063672 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7" pod "937292f6-aede-414c-a73d-a05ae812e483" (UID: "937292f6-aede-414c-a73d-a05ae812e483"). InnerVolumeSpecName "kube-api-access-9flk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:09.163657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.163625 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flk7\" (UniqueName: \"kubernetes.io/projected/937292f6-aede-414c-a73d-a05ae812e483-kube-api-access-9flk7\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:06:09.266310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.266268 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937292f6-aede-414c-a73d-a05ae812e483" path="/var/lib/kubelet/pods/937292f6-aede-414c-a73d-a05ae812e483/volumes" Oct 30 01:06:09.754122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.754084 5548 generic.go:356] "Generic (PLEG): container finished" podID="937292f6-aede-414c-a73d-a05ae812e483" containerID="53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30" exitCode=137 Oct 30 01:06:09.754276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.754128 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-bc628c" Oct 30 01:06:09.754276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.754188 5548 scope.go:122] "RemoveContainer" containerID="53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30" Oct 30 01:06:09.764202 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.763927 5548 scope.go:122] "RemoveContainer" containerID="53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30" Oct 30 01:06:09.764284 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:09.764250 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30\": container with ID starting with 53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30 not found: ID does not exist" containerID="53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30" Oct 30 01:06:09.764284 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:09.764272 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30"} err="failed to get container status \"53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30\": rpc error: code = NotFound desc = could not find container \"53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30\": container with ID starting with 53650651c5f086435d7e928ea4217d5bafe6944a38687a54f02af8cf5f919c30 not found: ID does not exist" Oct 30 01:06:19.032145 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:19.032113 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw"] Oct 30 01:06:19.041749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:19.041725 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872946f5-hlqlw"] Oct 30 01:06:19.266668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:19.266638 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2a19cb-dd21-49d3-8047-a0630e150928" path="/var/lib/kubelet/pods/1c2a19cb-dd21-49d3-8047-a0630e150928/volumes" Oct 30 01:06:24.421302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.421272 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:06:24.421987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.421971 5548 cpu_manager.go:414] "containerMap: removing container" podUID="937292f6-aede-414c-a73d-a05ae812e483" containerName="test-curl-8753-bc628c" Oct 30 01:06:24.421987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.421988 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937292f6-aede-414c-a73d-a05ae812e483" containerName="test-curl-8753-bc628c" Oct 30 01:06:24.422112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.422101 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="937292f6-aede-414c-a73d-a05ae812e483" containerName="test-curl-8753-bc628c" Oct 30 01:06:24.425712 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.425692 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.450160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.450129 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-288da6-serving-cert\"" Oct 30 01:06:24.456532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.456503 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:06:24.525535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.525502 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.525706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.525665 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.525768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.525714 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.525812 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.525798 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.628955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.628036 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.628955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.628211 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.628955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.628278 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.636569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.629477 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.636569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.629495 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.644381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.643801 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.644930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.644900 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.647597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.647473 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\") pod \"maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:24.741043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:24.740967 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:25.049900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:25.049836 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:06:25.051043 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:06:25.051015 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5daaafb_a3e0_4f02_9409_9df2179a7671.slice/crio-dfd0575b666c0601aaa465fc6a03c4e4a66c1ec7916262e55043bf6a2ba58542 WatchSource:0}: Error finding container dfd0575b666c0601aaa465fc6a03c4e4a66c1ec7916262e55043bf6a2ba58542: Status 404 returned error can't find the container with id dfd0575b666c0601aaa465fc6a03c4e4a66c1ec7916262e55043bf6a2ba58542 Oct 30 01:06:25.436059 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:25.436021 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:06:25.984357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:25.984317 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" event={"ID":"c5daaafb-a3e0-4f02-9409-9df2179a7671","Type":"ContainerStarted","Data":"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a"} Oct 30 01:06:25.984626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:25.984398 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" event={"ID":"c5daaafb-a3e0-4f02-9409-9df2179a7671","Type":"ContainerStarted","Data":"dfd0575b666c0601aaa465fc6a03c4e4a66c1ec7916262e55043bf6a2ba58542"} Oct 30 01:06:25.984626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:25.984557 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:26.018310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:26.018262 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" podStartSLOduration=1.666129814 podStartE2EDuration="2.018246742s" totalImagesPullingTime="352.116928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:26.017809613 +0000 UTC m=+1376.860370218" watchObservedRunningTime="2026-10-30 01:06:26.018246742 +0000 UTC m=+1376.860807386" Oct 30 01:06:30.466020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:30.465964 5548 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:30.693762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:30.693708 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:32.017599 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:32.017571 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:06:42.663510 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.663479 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:06:42.670084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.670063 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.684127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.684107 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a3204e-serving-cert\"" Oct 30 01:06:42.706201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.706168 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:06:42.799788 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.799753 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.799998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.799815 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.799998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.799986 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgpf\" (UniqueName: \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.800135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.800088 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.903085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.902347 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.903085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.902491 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgpf\" (UniqueName: \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.903085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.902548 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.903085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.902807 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.903814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.903672 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.910036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.910008 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.924503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.924425 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.927240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.927205 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgpf\" (UniqueName: \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\") pod \"maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:42.984194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:42.984147 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:43.315966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:43.315930 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:06:43.316997 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:06:43.316963 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aa4cc0_0ef7_426f_a35d_8204ff9d98f3.slice/crio-4eaa461d897f0286f4eb77f1944ac0b36a87c89a9ed7f55bfa787a680594d02d WatchSource:0}: Error finding container 4eaa461d897f0286f4eb77f1944ac0b36a87c89a9ed7f55bfa787a680594d02d: Status 404 returned error can't find the container with id 4eaa461d897f0286f4eb77f1944ac0b36a87c89a9ed7f55bfa787a680594d02d Oct 30 01:06:43.318642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:43.318622 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:06:43.355275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:43.355239 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" event={"ID":"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3","Type":"ContainerStarted","Data":"4eaa461d897f0286f4eb77f1944ac0b36a87c89a9ed7f55bfa787a680594d02d"} Oct 30 01:06:44.371278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:44.371230 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" event={"ID":"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3","Type":"ContainerStarted","Data":"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2"} Oct 30 01:06:44.371719 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:44.371686 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:44.408576 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:44.408434 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" podStartSLOduration=2.042012164 podStartE2EDuration="2.408416293s" totalImagesPullingTime="366.404129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:44.406995392 +0000 UTC m=+1395.249555984" watchObservedRunningTime="2026-10-30 01:06:44.408416293 +0000 UTC m=+1395.250976895" Oct 30 01:06:50.384340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:06:50.384313 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:06:55.451444 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:06:55.451403 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:07:03.591518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.591485 5548 generic.go:356] "Generic (PLEG): container finished" podID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerID="67d5dcdb1962a743d7aa7f2225c50135a4528b37a1a78dfa12379e9050a4b317" exitCode=0 Oct 30 01:07:03.591988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.591552 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerDied","Data":"67d5dcdb1962a743d7aa7f2225c50135a4528b37a1a78dfa12379e9050a4b317"} Oct 30 01:07:03.592046 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.592019 5548 scope.go:122] "RemoveContainer" containerID="67d5dcdb1962a743d7aa7f2225c50135a4528b37a1a78dfa12379e9050a4b317" Oct 30 01:07:03.593790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.593755 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:07:03.593952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.593847 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:07:03.593952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:03.593924 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:07:04.601928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:04.601896 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerStarted","Data":"781406937bc9ba52c0a95e7f5c280b81b54f086a3429b226c07dae0d9b172f9f"} Oct 30 01:07:04.658837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:04.657140 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:07:04.658837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:04.657281 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:04.658837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:04.657337 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:05.649179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:05.649139 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:05.649657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:05.649194 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:06.648321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:06.648289 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:06.648496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:06.648342 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:07.649382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:07.649347 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:07.649787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:07.649412 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:08.659377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:08.659341 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:08.660003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:08.659971 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:09.655885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:09.649609 5548 patch_prober.go:29] interesting pod/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:07:09.655885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:09.649678 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 01:07:10.718199 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:10.718164 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:07:10.719911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:10.718911 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:07:10.721038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:10.721013 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:07:25.434732 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:25.434689 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:07:29.747005 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:29.746971 5548 scope.go:122] "RemoveContainer" containerID="7941f7d5992da1f5827779c8f1489c3b69bf6b854fb0a1f254ce29d620447ebb" Oct 30 01:07:40.584669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.584588 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:07:40.588728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.588712 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.622452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.622410 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:07:40.665837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.665805 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb3370cb-serving-cert\"" Oct 30 01:07:40.682290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.682259 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.682417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.682353 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.682460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.682449 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlhd\" (UniqueName: \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.682509 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.682495 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.792539 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.792486 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.792735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.792567 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.792735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.792653 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlhd\" (UniqueName: \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.792735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.792704 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.804040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.793579 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.808372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.808335 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.811513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.811482 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.825308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.825280 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlhd\" (UniqueName: \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:40.903164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:40.903131 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:41.209034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:41.209005 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:07:41.210016 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:07:41.209992 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891c5644_9720_4ea8_b01c_7bb81f26fefd.slice/crio-15992ad51a8a28cf201a3cf539026f4450452467074ba18f8a9afa250b09359a WatchSource:0}: Error finding container 15992ad51a8a28cf201a3cf539026f4450452467074ba18f8a9afa250b09359a: Status 404 returned error can't find the container with id 15992ad51a8a28cf201a3cf539026f4450452467074ba18f8a9afa250b09359a Oct 30 01:07:42.067978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:42.056955 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" event={"ID":"891c5644-9720-4ea8-b01c-7bb81f26fefd","Type":"ContainerStarted","Data":"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b"} Oct 30 01:07:42.067978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:42.057024 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" event={"ID":"891c5644-9720-4ea8-b01c-7bb81f26fefd","Type":"ContainerStarted","Data":"15992ad51a8a28cf201a3cf539026f4450452467074ba18f8a9afa250b09359a"} Oct 30 01:07:42.067978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:42.057978 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:42.097514 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:42.097448 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" podStartSLOduration=1.651379733 podStartE2EDuration="2.097428821s" totalImagesPullingTime="446.049088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:42.095917013 +0000 UTC m=+1452.938477627" watchObservedRunningTime="2026-10-30 01:07:42.097428821 +0000 UTC m=+1452.939989415" Oct 30 01:07:45.828751 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.828714 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:07:45.832420 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.832404 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.861757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.861721 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:07:45.875358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.875321 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7w4\" (UniqueName: \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.875553 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.875538 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.875640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.875575 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.875837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.875812 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.934809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.934777 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-262230-serving-cert\"" Oct 30 01:07:45.977692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.977654 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7w4\" (UniqueName: \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.977900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.977761 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.977900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.977788 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.977900 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:45.977874 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-serving-cert: secret "maas-api-e2e-shared-a-w1-262230-serving-cert" not found Oct 30 01:07:45.978028 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:45.977946 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls podName:16f9868e-83be-4b11-a343-4b0609e143b5 nodeName:}" failed. No retries permitted until 2026-10-30 01:07:46.477931093 +0000 UTC m=+1457.320491664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls") pod "maas-api-e2e-shared-a-w1-262230-677578987f-nl788" (UID: "16f9868e-83be-4b11-a343-4b0609e143b5") : secret "maas-api-e2e-shared-a-w1-262230-serving-cert" not found Oct 30 01:07:45.978028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.977985 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.978564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.978542 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.980635 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.980610 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:45.995277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:45.995251 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7w4\" (UniqueName: \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:46.486613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:46.486582 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:46.489495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:46.489461 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-262230-677578987f-nl788\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:46.747454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:46.747351 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:47.076777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:47.076739 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:07:47.102993 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:47.102958 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" event={"ID":"16f9868e-83be-4b11-a343-4b0609e143b5","Type":"ContainerStarted","Data":"450ced9b33a9ddaf51e8a3a4807fec5158bb0cc030f5509198f47040c7f61cd7"} Oct 30 01:07:48.112486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:48.112451 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" event={"ID":"16f9868e-83be-4b11-a343-4b0609e143b5","Type":"ContainerStarted","Data":"6c89cdc310ed786327deb5cf1d6033ac651f5c165eb5171de1a8a8cafe409af0"} Oct 30 01:07:48.112968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:48.112546 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:48.149141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:48.149091 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" podStartSLOduration=2.800856379 podStartE2EDuration="3.149073703s" totalImagesPullingTime="348.217324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:48.146965618 +0000 UTC m=+1458.989526220" watchObservedRunningTime="2026-10-30 01:07:48.149073703 +0000 UTC m=+1458.991634295" Oct 30 01:07:49.077563 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:49.077534 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:07:52.646825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.646787 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 01:07:52.647368 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.647023 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" containerID="cri-o://8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2" gracePeriod=30 Oct 30 01:07:52.657965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.657935 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 01:07:52.658400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.658345 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" containerID="cri-o://6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77" gracePeriod=30 Oct 30 01:07:52.666951 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.664350 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 01:07:52.666951 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.664581 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" containerID="cri-o://57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108" gracePeriod=30 Oct 30 01:07:52.902058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.901967 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 01:07:52.902240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.902207 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" containerID="cri-o://e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92" gracePeriod=30 Oct 30 01:07:52.905768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.905732 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:07:52.906002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.905975 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" containerID="cri-o://5ad60e5eedadb8cf274890b88069a53ede130aa0d8e7cb05b63c5897f54288d3" gracePeriod=30 Oct 30 01:07:52.909751 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.909725 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 01:07:52.909986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:52.909964 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" containerID="cri-o://f6855283695e146196fca24d65a59f459208408991c5ffdb246f44b4cd93d2a3" gracePeriod=30 Oct 30 01:07:54.127797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:54.127768 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:07:55.433041 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:55.433006 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:07:58.122918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.122843 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 01:07:58.133016 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.132781 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 01:07:58.134912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.134715 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 01:07:58.193310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.193288 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 01:07:58.201277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.201152 5548 generic.go:356] "Generic (PLEG): container finished" podID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerID="e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92" exitCode=0 Oct 30 01:07:58.201277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.201270 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" event={"ID":"f86979f2-c453-41b0-aa44-efbae0f70fb0","Type":"ContainerDied","Data":"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92"} Oct 30 01:07:58.201461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.201324 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" event={"ID":"f86979f2-c453-41b0-aa44-efbae0f70fb0","Type":"ContainerDied","Data":"f1f6a759b4a17d9ff2b6b659c610e0ede198d0fadf0e1fcd39af401101b5935e"} Oct 30 01:07:58.201461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.201354 5548 scope.go:122] "RemoveContainer" containerID="e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92" Oct 30 01:07:58.201560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.201489 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd" Oct 30 01:07:58.212789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.212749 5548 generic.go:356] "Generic (PLEG): container finished" podID="de158740-04aa-40fd-ad73-2f1917a9525c" containerID="5ad60e5eedadb8cf274890b88069a53ede130aa0d8e7cb05b63c5897f54288d3" exitCode=0 Oct 30 01:07:58.213285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.213165 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" event={"ID":"de158740-04aa-40fd-ad73-2f1917a9525c","Type":"ContainerDied","Data":"5ad60e5eedadb8cf274890b88069a53ede130aa0d8e7cb05b63c5897f54288d3"} Oct 30 01:07:58.216310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.216281 5548 generic.go:356] "Generic (PLEG): container finished" podID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerID="6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77" exitCode=0 Oct 30 01:07:58.216611 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.216588 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" Oct 30 01:07:58.216796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.216766 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" event={"ID":"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8","Type":"ContainerDied","Data":"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77"} Oct 30 01:07:58.217185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.216816 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn" event={"ID":"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8","Type":"ContainerDied","Data":"45b75a349bdced25be322e74d20a46434db6c4592cc648f018cfb09b75bb4205"} Oct 30 01:07:58.218766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.218742 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerID="57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108" exitCode=0 Oct 30 01:07:58.219798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.219777 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" event={"ID":"4e735e85-42d3-4caa-a9be-33445bfa6d0b","Type":"ContainerDied","Data":"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108"} Oct 30 01:07:58.220027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.220006 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" event={"ID":"4e735e85-42d3-4caa-a9be-33445bfa6d0b","Type":"ContainerDied","Data":"117af7768ee4813dbf63a40811f9854a48f981c7c9f0513a1474836fd9911406"} Oct 30 01:07:58.220234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.220217 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8" Oct 30 01:07:58.226238 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.226216 5548 generic.go:356] "Generic (PLEG): container finished" podID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerID="8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2" exitCode=0 Oct 30 01:07:58.226432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.226412 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" event={"ID":"9d569248-d28e-4f24-8b97-dad688dde3ca","Type":"ContainerDied","Data":"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2"} Oct 30 01:07:58.226635 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.226615 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" event={"ID":"9d569248-d28e-4f24-8b97-dad688dde3ca","Type":"ContainerDied","Data":"0f14b4b25640fc5734d81e613a120be9130c851e1ef1cbedbc39bd15fbb7961f"} Oct 30 01:07:58.226822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.226798 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc" Oct 30 01:07:58.229197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.229177 5548 generic.go:356] "Generic (PLEG): container finished" podID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerID="f6855283695e146196fca24d65a59f459208408991c5ffdb246f44b4cd93d2a3" exitCode=0 Oct 30 01:07:58.229429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.229387 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" event={"ID":"31fc7b71-ddec-475b-921c-4c4708bc7ef7","Type":"ContainerDied","Data":"f6855283695e146196fca24d65a59f459208408991c5ffdb246f44b4cd93d2a3"} Oct 30 01:07:58.237102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.237007 5548 scope.go:122] "RemoveContainer" containerID="e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92" Oct 30 01:07:58.237779 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:58.237451 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92\": container with ID starting with e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92 not found: ID does not exist" containerID="e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92" Oct 30 01:07:58.237779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.237490 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92"} err="failed to get container status \"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92\": rpc error: code = NotFound desc = could not find container \"e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92\": container with ID starting with e41cca022f749de940e45c962396b308db1860b769ce6fad31ae11b1412b5a92 not found: ID does not exist" Oct 30 01:07:58.237779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.237514 5548 scope.go:122] "RemoveContainer" containerID="6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77" Oct 30 01:07:58.271359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.271333 5548 scope.go:122] "RemoveContainer" containerID="6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77" Oct 30 01:07:58.271790 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:58.271763 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77\": container with ID starting with 6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77 not found: ID does not exist" containerID="6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77" Oct 30 01:07:58.271902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.271804 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77"} err="failed to get container status \"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77\": rpc error: code = NotFound desc = could not find container \"6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77\": container with ID starting with 6348330bd0a3a96990086ab4ada8d7655a086e19e6ba23155923eb2141146c77 not found: ID does not exist" Oct 30 01:07:58.271902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.271829 5548 scope.go:122] "RemoveContainer" containerID="57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108" Oct 30 01:07:58.279313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.279289 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:07:58.285952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.285934 5548 scope.go:122] "RemoveContainer" containerID="57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108" Oct 30 01:07:58.286385 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:58.286356 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108\": container with ID starting with 57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108 not found: ID does not exist" containerID="57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108" Oct 30 01:07:58.286513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.286395 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108"} err="failed to get container status \"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108\": rpc error: code = NotFound desc = could not find container \"57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108\": container with ID starting with 57c70074b9cebce41f8a67d8854f5b8f4a242ae89557140fb1edf740db46b108 not found: ID does not exist" Oct 30 01:07:58.286513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.286412 5548 scope.go:122] "RemoveContainer" containerID="8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2" Oct 30 01:07:58.295727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295700 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.295893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295748 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.295893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295780 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.295893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295807 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.295893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295888 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295929 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.295962 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296015 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296052 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296103 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296145 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296190 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296226 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296265 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296294 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296311 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296333 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296357 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296375 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296392 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296457 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296476 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296500 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296536 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296574 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296595 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\") pod \"f86979f2-c453-41b0-aa44-efbae0f70fb0\" (UID: \"f86979f2-c453-41b0-aa44-efbae0f70fb0\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296649 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296686 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296815 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296843 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296957 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.296975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.296916 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297129 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297189 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297227 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297240 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297314 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\") pod \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\" (UID: \"4e735e85-42d3-4caa-a9be-33445bfa6d0b\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297344 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\") pod \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\" (UID: \"15da3c6a-3e23-4a89-b1a8-29b6f4c919f8\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297385 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\") pod \"9d569248-d28e-4f24-8b97-dad688dde3ca\" (UID: \"9d569248-d28e-4f24-8b97-dad688dde3ca\") " Oct 30 01:07:58.297707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.297495 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.298319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298297 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.298389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298327 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86979f2-c453-41b0-aa44-efbae0f70fb0-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.298389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298343 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.298389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298357 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.298618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298598 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.299237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298837 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.299237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.298935 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.299647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.299619 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.299779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.299761 5548 scope.go:122] "RemoveContainer" containerID="8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2" Oct 30 01:07:58.300159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.300137 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.300249 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.300169 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.300665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.300640 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.300819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.300797 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 01:07:58.301603 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:07:58.301572 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2\": container with ID starting with 8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2 not found: ID does not exist" containerID="8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2" Oct 30 01:07:58.301710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.301600 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2"} err="failed to get container status \"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2\": rpc error: code = NotFound desc = could not find container \"8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2\": container with ID starting with 8a73198b41a71df5b01084d687d5a2a505fc30462f996add028234b38e5453d2 not found: ID does not exist" Oct 30 01:07:58.303396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.303106 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.303957 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304031 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304258 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304326 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304476 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304595 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304681 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.304807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.304781 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.305588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.305544 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.306059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.305747 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.306059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.305871 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.306059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.305943 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "kube-api-access-hlmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.306059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.306001 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.306424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.306388 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.306499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.306437 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.307315 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.307291 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.307459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.307442 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "kube-api-access-qvcx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.308377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.308352 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.309176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.309154 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.309265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.309176 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4" pod "9d569248-d28e-4f24-8b97-dad688dde3ca" (UID: "9d569248-d28e-4f24-8b97-dad688dde3ca"). InnerVolumeSpecName "kube-api-access-w9zt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.309265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.309186 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.309265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.309197 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token" pod "f86979f2-c453-41b0-aa44-efbae0f70fb0" (UID: "f86979f2-c453-41b0-aa44-efbae0f70fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.309375 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.309223 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4" pod "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" (UID: "15da3c6a-3e23-4a89-b1a8-29b6f4c919f8"). InnerVolumeSpecName "kube-api-access-p9wd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.310619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.310593 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo" pod "4e735e85-42d3-4caa-a9be-33445bfa6d0b" (UID: "4e735e85-42d3-4caa-a9be-33445bfa6d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.399488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.399458 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.399778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.399551 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.399778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.399740 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.400309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.400286 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.400804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.400684 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.401057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.401038 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.401486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.401374 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.401655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.401640 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.406081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.406000 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408294 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408379 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408415 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408526 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408485 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408526 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408520 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408557 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408605 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408647 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408791 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408732 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408822 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.408921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408893 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\") pod \"de158740-04aa-40fd-ad73-2f1917a9525c\" (UID: \"de158740-04aa-40fd-ad73-2f1917a9525c\") " Oct 30 01:07:58.408921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408850 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.409050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.408931 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\") pod \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\" (UID: \"31fc7b71-ddec-475b-921c-4c4708bc7ef7\") " Oct 30 01:07:58.409200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.409173 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.409587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.409556 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "kube-api-access-2xjtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.410272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.410104 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.410366 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.410305 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.410624 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.410600 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.411364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411335 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411875 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wd4\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-kube-api-access-p9wd4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411914 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411941 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411956 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411973 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.411987 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412009 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412024 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412038 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412053 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412073 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412088 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412103 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412116 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmdk\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-kube-api-access-hlmdk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412143 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412166 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412177 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412185 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcx2\" (UniqueName: \"kubernetes.io/projected/4e735e85-42d3-4caa-a9be-33445bfa6d0b-kube-api-access-qvcx2\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412196 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412211 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412220 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d569248-d28e-4f24-8b97-dad688dde3ca-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412225 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412233 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412278 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412298 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412320 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e735e85-42d3-4caa-a9be-33445bfa6d0b-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412335 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412350 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjtp\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-kube-api-access-2xjtp\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412364 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412386 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d569248-d28e-4f24-8b97-dad688dde3ca-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412593 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d569248-d28e-4f24-8b97-dad688dde3ca-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412609 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412623 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zt4\" (UniqueName: \"kubernetes.io/projected/9d569248-d28e-4f24-8b97-dad688dde3ca-kube-api-access-w9zt4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.412610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412642 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412655 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412670 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e735e85-42d3-4caa-a9be-33445bfa6d0b-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412816 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86979f2-c453-41b0-aa44-efbae0f70fb0-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412837 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412851 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86979f2-c453-41b0-aa44-efbae0f70fb0-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.412894 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.413482 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.413480 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:58.414728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.414280 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.421227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.421204 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "kube-api-access-7jzlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:58.425632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.425590 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo" pod "de158740-04aa-40fd-ad73-2f1917a9525c" (UID: "de158740-04aa-40fd-ad73-2f1917a9525c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.427400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.427371 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:07:58.427600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.427583 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo" pod "31fc7b71-ddec-475b-921c-4c4708bc7ef7" (UID: "31fc7b71-ddec-475b-921c-4c4708bc7ef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:07:58.521191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521155 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de158740-04aa-40fd-ad73-2f1917a9525c-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521192 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521206 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521218 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de158740-04aa-40fd-ad73-2f1917a9525c-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521231 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521242 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521254 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fc7b71-ddec-475b-921c-4c4708bc7ef7-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521268 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521281 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de158740-04aa-40fd-ad73-2f1917a9525c-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521293 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fc7b71-ddec-475b-921c-4c4708bc7ef7-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.521799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.521305 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzlw\" (UniqueName: \"kubernetes.io/projected/de158740-04aa-40fd-ad73-2f1917a9525c-kube-api-access-7jzlw\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:07:58.613560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.613506 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 01:07:58.636620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.634937 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-cr9pd"] Oct 30 01:07:58.650067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.650035 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 01:07:58.651779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.651751 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-2vdm8"] Oct 30 01:07:58.687124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.686922 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 01:07:58.689426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.689319 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-x64vc"] Oct 30 01:07:58.717713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.717675 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 01:07:58.723577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:58.722900 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-flxfn"] Oct 30 01:07:59.238095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.238063 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" event={"ID":"31fc7b71-ddec-475b-921c-4c4708bc7ef7","Type":"ContainerDied","Data":"e35aa030be75d2c5ba6fd6a1a2de0be0b3f127a4e9a02c8e25e17fa9951842ce"} Oct 30 01:07:59.238566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.238072 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv" Oct 30 01:07:59.238566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.238165 5548 scope.go:122] "RemoveContainer" containerID="f6855283695e146196fca24d65a59f459208408991c5ffdb246f44b4cd93d2a3" Oct 30 01:07:59.240688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.240666 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" Oct 30 01:07:59.240800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.240673 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg" event={"ID":"de158740-04aa-40fd-ad73-2f1917a9525c","Type":"ContainerDied","Data":"ad5303bb01d8625a8b83705d4ab96c8e813dde7c42c98912d338dbe72f6c6d8e"} Oct 30 01:07:59.250065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.250015 5548 scope.go:122] "RemoveContainer" containerID="5ad60e5eedadb8cf274890b88069a53ede130aa0d8e7cb05b63c5897f54288d3" Oct 30 01:07:59.265310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.265282 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" path="/var/lib/kubelet/pods/15da3c6a-3e23-4a89-b1a8-29b6f4c919f8/volumes" Oct 30 01:07:59.265821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.265800 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" path="/var/lib/kubelet/pods/4e735e85-42d3-4caa-a9be-33445bfa6d0b/volumes" Oct 30 01:07:59.266362 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.266342 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" path="/var/lib/kubelet/pods/9d569248-d28e-4f24-8b97-dad688dde3ca/volumes" Oct 30 01:07:59.267161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.267139 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" path="/var/lib/kubelet/pods/f86979f2-c453-41b0-aa44-efbae0f70fb0/volumes" Oct 30 01:07:59.285589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.285565 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:07:59.288899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.288876 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-d52rg"] Oct 30 01:07:59.317583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.317555 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 01:07:59.323904 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:07:59.323875 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-phpkv"] Oct 30 01:08:00.535944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.535906 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:08:00.536799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536777 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536801 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536815 5548 cpu_manager.go:414] "containerMap: removing container" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536823 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536853 5548 cpu_manager.go:414] "containerMap: removing container" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536879 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536893 5548 cpu_manager.go:414] "containerMap: removing container" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536900 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536943 5548 cpu_manager.go:414] "containerMap: removing container" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" Oct 30 01:08:00.536964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536949 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536973 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.536980 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537169 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e735e85-42d3-4caa-a9be-33445bfa6d0b" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537185 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d569248-d28e-4f24-8b97-dad688dde3ca" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537201 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537214 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537232 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="15da3c6a-3e23-4a89-b1a8-29b6f4c919f8" containerName="istio-proxy" Oct 30 01:08:00.537397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.537245 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86979f2-c453-41b0-aa44-efbae0f70fb0" containerName="istio-proxy" Oct 30 01:08:00.542922 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.542901 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.575662 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.575628 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:08:00.577093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.577071 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7e233f-serving-cert\"" Oct 30 01:08:00.667353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.667321 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.667524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.667389 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grh88\" (UniqueName: \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.667524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.667483 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.667658 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.667639 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.770933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.770899 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.771104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.771046 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.771104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.771090 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grh88\" (UniqueName: \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.771174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.771135 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.771278 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:00.771262 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-serving-cert: secret "maas-api-e2e-inf-b-7e233f-serving-cert" not found Oct 30 01:08:00.771366 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:00.771337 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls podName:e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19 nodeName:}" failed. No retries permitted until 2026-10-30 01:08:01.27131612 +0000 UTC m=+1472.113876706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls") pod "maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" (UID: "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19") : secret "maas-api-e2e-inf-b-7e233f-serving-cert" not found Oct 30 01:08:00.771794 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.771767 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.773781 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.773751 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:00.791084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:00.791019 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grh88\" (UniqueName: \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:01.264687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.264655 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fc7b71-ddec-475b-921c-4c4708bc7ef7" path="/var/lib/kubelet/pods/31fc7b71-ddec-475b-921c-4c4708bc7ef7/volumes" Oct 30 01:08:01.265174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.265160 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de158740-04aa-40fd-ad73-2f1917a9525c" path="/var/lib/kubelet/pods/de158740-04aa-40fd-ad73-2f1917a9525c/volumes" Oct 30 01:08:01.276910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.276883 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:01.279372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.279348 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:01.460953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.460902 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:01.779746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:01.779714 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:08:01.781122 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:01.781094 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f2c4e0_7e4b_4c5b_ac8a_7fb12639ad19.slice/crio-75cad0de5abe715e625cdfd2b4e853b67df099c3a90eeeeb5a33e9649a2c6f5a WatchSource:0}: Error finding container 75cad0de5abe715e625cdfd2b4e853b67df099c3a90eeeeb5a33e9649a2c6f5a: Status 404 returned error can't find the container with id 75cad0de5abe715e625cdfd2b4e853b67df099c3a90eeeeb5a33e9649a2c6f5a Oct 30 01:08:02.269647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:02.269620 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" event={"ID":"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19","Type":"ContainerStarted","Data":"75cad0de5abe715e625cdfd2b4e853b67df099c3a90eeeeb5a33e9649a2c6f5a"} Oct 30 01:08:03.277830 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:03.277767 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" event={"ID":"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19","Type":"ContainerStarted","Data":"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224"} Oct 30 01:08:03.278269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:03.277847 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:03.313403 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:03.313340 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" podStartSLOduration=2.92609444 podStartE2EDuration="3.313326461s" totalImagesPullingTime="387.232021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:03.312648039 +0000 UTC m=+1474.155208632" watchObservedRunningTime="2026-10-30 01:08:03.313326461 +0000 UTC m=+1474.155887101" Oct 30 01:08:03.968738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:03.968706 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:08:03.974150 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:03.974126 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.003152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.002910 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:08:04.004156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.004133 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da60de-openshift-default-dockercfg-55g65\"" Oct 30 01:08:04.020302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020263 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020350 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020438 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020490 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqf6t\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020588 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020622 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020653 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.020881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.020831 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.021031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.021010 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.155944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.139805 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.167977 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168101 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168138 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqf6t\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168176 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168217 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168270 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168407 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.168494 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.169990 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.170229 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.172156 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.173884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.172693 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.178168 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.178143 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.198886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.179671 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.199118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.196449 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.217216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.217187 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.220416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.220362 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqf6t\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\") pod \"e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.310381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.310350 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:04.652719 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:04.652684 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c81780_34d0_4a2c_be09_2b6e3eefe525.slice/crio-81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422 WatchSource:0}: Error finding container 81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422: Status 404 returned error can't find the container with id 81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422 Oct 30 01:08:04.653422 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.653396 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:08:04.656063 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.656027 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:04.656182 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.656112 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:04.656182 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:04.656155 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:05.306770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:05.306731 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerStarted","Data":"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0"} Oct 30 01:08:05.306972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:05.306826 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerStarted","Data":"81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422"} Oct 30 01:08:05.311104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:05.311083 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:05.317022 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:05.317001 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:05.358589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:05.358528 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podStartSLOduration=2.358509353 podStartE2EDuration="2.358509353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:05.354323659 +0000 UTC m=+1476.196884254" watchObservedRunningTime="2026-10-30 01:08:05.358509353 +0000 UTC m=+1476.201069947" Oct 30 01:08:06.313696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:06.313670 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:06.314716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:06.314698 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:08:09.306989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:09.306464 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:08:10.298697 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:10.298656 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:08:10.298994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:10.298966 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" containerID="cri-o://53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17" gracePeriod=30 Oct 30 01:08:13.633630 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:13.633594 5548 patch_prober.go:29] interesting pod/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:08:13.634054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:13.633646 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:08:15.672918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.662849 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:08:15.770963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.770924 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.770990 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771085 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771136 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771259 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771310 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771350 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771386 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:15.771537 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771449 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771537 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771464 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:15.771537 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771509 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\") pod \"bd915a47-3112-46bd-ba50-1bd19a846192\" (UID: \"bd915a47-3112-46bd-ba50-1bd19a846192\") " Oct 30 01:08:15.771770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.771747 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:15.772133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772106 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:15.772366 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772349 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.772452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772368 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.772452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772381 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd915a47-3112-46bd-ba50-1bd19a846192-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.772452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772392 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.772611 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.772548 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:15.773605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.773579 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:15.773801 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.773779 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:15.773873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.773783 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "kube-api-access-pfmg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:15.774318 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.774295 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy" pod "bd915a47-3112-46bd-ba50-1bd19a846192" (UID: "bd915a47-3112-46bd-ba50-1bd19a846192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:15.874232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.874134 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.874232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.874172 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd915a47-3112-46bd-ba50-1bd19a846192-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.874232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.874186 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfmg4\" (UniqueName: \"kubernetes.io/projected/bd915a47-3112-46bd-ba50-1bd19a846192-kube-api-access-pfmg4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.874232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.874201 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:15.874232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:15.874213 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd915a47-3112-46bd-ba50-1bd19a846192-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:16.413288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.413239 5548 generic.go:356] "Generic (PLEG): container finished" podID="bd915a47-3112-46bd-ba50-1bd19a846192" containerID="53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17" exitCode=0 Oct 30 01:08:16.413645 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.413625 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerDied","Data":"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17"} Oct 30 01:08:16.413783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.413768 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" event={"ID":"bd915a47-3112-46bd-ba50-1bd19a846192","Type":"ContainerDied","Data":"d550c85439f48c8229d7f0b23b1fc03906e5dc198602bb45f74a96df892dbc5c"} Oct 30 01:08:16.413931 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.413915 5548 scope.go:122] "RemoveContainer" containerID="53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17" Oct 30 01:08:16.414110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.414090 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db" Oct 30 01:08:16.458140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.458095 5548 scope.go:122] "RemoveContainer" containerID="5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad" Oct 30 01:08:16.481140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.480710 5548 scope.go:122] "RemoveContainer" containerID="53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17" Oct 30 01:08:16.481279 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:16.481199 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17\": container with ID starting with 53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17 not found: ID does not exist" containerID="53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17" Oct 30 01:08:16.481279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.481239 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17"} err="failed to get container status \"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17\": rpc error: code = NotFound desc = could not find container \"53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17\": container with ID starting with 53af2f4e2fca5c8e1c19d219dc87e9227128cde642e77877ea6d1851d4d45d17 not found: ID does not exist" Oct 30 01:08:16.481423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.481282 5548 scope.go:122] "RemoveContainer" containerID="5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad" Oct 30 01:08:16.482206 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:16.482137 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad\": container with ID starting with 5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad not found: ID does not exist" containerID="5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad" Oct 30 01:08:16.482206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.482179 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad"} err="failed to get container status \"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad\": rpc error: code = NotFound desc = could not find container \"5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad\": container with ID starting with 5a1c20674f73ccdd3dbb3297db62ae60ad4c0545cfb367abca11227b83c616ad not found: ID does not exist" Oct 30 01:08:16.491919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.491589 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:08:16.502207 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:16.502177 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-17d855-openshift-default-599df4fbdf-rk5db"] Oct 30 01:08:17.326886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:17.323163 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" path="/var/lib/kubelet/pods/bd915a47-3112-46bd-ba50-1bd19a846192/volumes" Oct 30 01:08:25.090829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.090794 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8768-37fb86"] Oct 30 01:08:25.091854 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.091833 5548 cpu_manager.go:414] "containerMap: removing container" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.092009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.091874 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.092009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.091909 5548 cpu_manager.go:414] "containerMap: removing container" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.092009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.091917 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.092254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.092066 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.092254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.092085 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd915a47-3112-46bd-ba50-1bd19a846192" containerName="istio-proxy" Oct 30 01:08:25.096459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.096434 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:25.117228 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.117002 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:08:25.118937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.118910 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-37fb86"] Oct 30 01:08:25.176954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.176909 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6l42\" (UniqueName: \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\") pod \"test-curl-8768-37fb86\" (UID: \"9cfe06a3-e6ba-4246-8265-c257a99e985f\") " pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:25.278402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.278369 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6l42\" (UniqueName: \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\") pod \"test-curl-8768-37fb86\" (UID: \"9cfe06a3-e6ba-4246-8265-c257a99e985f\") " pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:25.296498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.296469 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6l42\" (UniqueName: \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\") pod \"test-curl-8768-37fb86\" (UID: \"9cfe06a3-e6ba-4246-8265-c257a99e985f\") " pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:25.421085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.421046 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:25.428556 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:25.428523 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:08:25.728397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:25.728369 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-37fb86"] Oct 30 01:08:25.729202 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:25.729174 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cfe06a3_e6ba_4246_8265_c257a99e985f.slice/crio-540075ae367356857a6e615f5307bd1f84d54634fd443fc0a2fddf7d5ce65733 WatchSource:0}: Error finding container 540075ae367356857a6e615f5307bd1f84d54634fd443fc0a2fddf7d5ce65733: Status 404 returned error can't find the container with id 540075ae367356857a6e615f5307bd1f84d54634fd443fc0a2fddf7d5ce65733 Oct 30 01:08:26.541889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:26.541818 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-37fb86" event={"ID":"9cfe06a3-e6ba-4246-8265-c257a99e985f","Type":"ContainerStarted","Data":"51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0"} Oct 30 01:08:26.542267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:26.541898 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-37fb86" event={"ID":"9cfe06a3-e6ba-4246-8265-c257a99e985f","Type":"ContainerStarted","Data":"540075ae367356857a6e615f5307bd1f84d54634fd443fc0a2fddf7d5ce65733"} Oct 30 01:08:26.573742 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:26.573690 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8768-37fb86" podStartSLOduration=1.5736764170000002 podStartE2EDuration="1.573676417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:26.573027588 +0000 UTC m=+1497.415588180" watchObservedRunningTime="2026-10-30 01:08:26.573676417 +0000 UTC m=+1497.416237010" Oct 30 01:08:27.391539 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:27.391478 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8768-37fb86"] Oct 30 01:08:27.396209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:27.396176 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8768-37fb86"] Oct 30 01:08:28.555096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:28.555054 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8768-37fb86" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" containerName="test-curl-8768-37fb86" containerID="cri-o://51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0" gracePeriod=2 Oct 30 01:08:28.566566 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:28.566536 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-37fb86\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:29.273460 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:29.273419 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-37fb86\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:29.545838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:29.545749 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:08:29.547157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:29.547134 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:08:30.174090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.174056 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:08:30.174960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.174939 5548 cpu_manager.go:414] "containerMap: removing container" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" containerName="test-curl-8768-37fb86" Oct 30 01:08:30.174960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.174960 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" containerName="test-curl-8768-37fb86" Oct 30 01:08:30.175158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.175144 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" containerName="test-curl-8768-37fb86" Oct 30 01:08:30.181348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.181326 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.211773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.208550 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:08:30.212945 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.212920 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-add1bb-openshift-default-dockercfg-hsjlp\"" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244275 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244403 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchw6\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244571 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244625 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244687 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244810 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244842 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244906 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.245598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.244962 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.346960 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347042 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347116 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchw6\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347293 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347355 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347420 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347599 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347625 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.347733 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.348288 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.359887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.348523 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.361457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.361427 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.365697 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.363669 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.365875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.364841 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.381072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.370896 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.381072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.372172 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.381072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.372696 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.422378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.422344 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchw6\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.499538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.499436 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:30.587517 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:30.585053 5548 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c81780_34d0_4a2c_be09_2b6e3eefe525.slice/crio-81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422\": RecentStats: unable to find data in memory cache]" Oct 30 01:08:30.813192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.813161 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:30.864517 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:30.864486 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac41a7_ccbe_46f9_9f19_8a6d4559fa40.slice/crio-c092b8e8096317aa364c2ddcb3e1408479efe7b66e620ccf13ca16062e6fcc71 WatchSource:0}: Error finding container c092b8e8096317aa364c2ddcb3e1408479efe7b66e620ccf13ca16062e6fcc71: Status 404 returned error can't find the container with id c092b8e8096317aa364c2ddcb3e1408479efe7b66e620ccf13ca16062e6fcc71 Oct 30 01:08:30.865829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.865808 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:08:30.868210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.868171 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:30.868298 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.868264 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:30.868335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.868315 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:08:30.958664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.958630 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\" (UniqueName: \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\") pod \"9cfe06a3-e6ba-4246-8265-c257a99e985f\" (UID: \"9cfe06a3-e6ba-4246-8265-c257a99e985f\") " Oct 30 01:08:30.960969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:30.960938 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42" pod "9cfe06a3-e6ba-4246-8265-c257a99e985f" (UID: "9cfe06a3-e6ba-4246-8265-c257a99e985f"). InnerVolumeSpecName "kube-api-access-h6l42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:31.061619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.061532 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6l42\" (UniqueName: \"kubernetes.io/projected/9cfe06a3-e6ba-4246-8265-c257a99e985f-kube-api-access-h6l42\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:31.269377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.269335 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cfe06a3-e6ba-4246-8265-c257a99e985f" path="/var/lib/kubelet/pods/9cfe06a3-e6ba-4246-8265-c257a99e985f/volumes" Oct 30 01:08:31.587055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.587018 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerStarted","Data":"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca"} Oct 30 01:08:31.587280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.587081 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerStarted","Data":"c092b8e8096317aa364c2ddcb3e1408479efe7b66e620ccf13ca16062e6fcc71"} Oct 30 01:08:31.588447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.588422 5548 generic.go:356] "Generic (PLEG): container finished" podID="9cfe06a3-e6ba-4246-8265-c257a99e985f" containerID="51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0" exitCode=137 Oct 30 01:08:31.588568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.588470 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-37fb86" Oct 30 01:08:31.588568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.588520 5548 scope.go:122] "RemoveContainer" containerID="51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0" Oct 30 01:08:31.598365 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.598347 5548 scope.go:122] "RemoveContainer" containerID="51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0" Oct 30 01:08:31.598654 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:31.598637 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0\": container with ID starting with 51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0 not found: ID does not exist" containerID="51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0" Oct 30 01:08:31.598705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.598663 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0"} err="failed to get container status \"51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0\": rpc error: code = NotFound desc = could not find container \"51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0\": container with ID starting with 51e94c9cf6ddde65f5882103b545ff1fad303abcbddc30fabed28fb2de95d2a0 not found: ID does not exist" Oct 30 01:08:31.626319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:31.626265 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podStartSLOduration=1.626249285 podStartE2EDuration="1.626249285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:31.622688884 +0000 UTC m=+1502.465249476" watchObservedRunningTime="2026-10-30 01:08:31.626249285 +0000 UTC m=+1502.468809963" Oct 30 01:08:32.499808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:32.499773 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:32.505678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:32.505656 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:32.597306 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:32.597281 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:32.598245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:32.598229 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:08:43.942506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.942470 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:08:43.950018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.949993 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:43.956835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.954449 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:08:43.975546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.975520 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:43.975702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.975654 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvkt\" (UniqueName: \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:43.975784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.975769 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:43.975892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.975878 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:43.996350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:43.996325 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da60de-serving-cert\"" Oct 30 01:08:44.077095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.077059 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.077297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.077177 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvkt\" (UniqueName: \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.077297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.077282 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.077421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.077364 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.077580 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:44.077557 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-serving-cert: secret "maas-api-e2e-api-b-da60de-serving-cert" not found Oct 30 01:08:44.077829 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:44.077796 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls podName:11da6415-854c-4b21-b4da-98c83edddd39 nodeName:}" failed. No retries permitted until 2026-10-30 01:08:44.577772281 +0000 UTC m=+1515.420332866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls") pod "maas-api-e2e-api-b-da60de-6857bf985-4jwkr" (UID: "11da6415-854c-4b21-b4da-98c83edddd39") : secret "maas-api-e2e-api-b-da60de-serving-cert" not found Oct 30 01:08:44.078312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.078287 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.080561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.080533 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.096235 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.096209 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvkt\" (UniqueName: \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.439259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.439224 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:08:44.445084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.445061 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.464137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.464111 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:08:44.468186 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.468160 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3204e\"/\"test-model-a3204e-kserve-self-signed-certs\"" Oct 30 01:08:44.468316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.468199 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a3204e\"/\"default-dockercfg-pwjgr\"" Oct 30 01:08:44.483549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.483522 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.484109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.484082 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qdk\" (UniqueName: \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.484301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.484277 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.484405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.484378 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.484476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.484405 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.484544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.484528 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.485988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.485968 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3204e\"/\"openshift-service-ca.crt\"" Oct 30 01:08:44.486099 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.486037 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a3204e\"/\"kube-root-ca.crt\"" Oct 30 01:08:44.586994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.586958 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587083 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587114 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587183 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587278 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587345 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qdk\" (UniqueName: \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587385 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.587748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587695 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.587748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587721 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.588016 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.587993 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.590112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.590047 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.590460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.590437 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.590571 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.590550 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") pod \"maas-api-e2e-api-b-da60de-6857bf985-4jwkr\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:44.603934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.603907 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qdk\" (UniqueName: \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\") pod \"test-model-a3204e-kserve-96df9444f-gxbvx\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.766480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.765667 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:44.867920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:44.867534 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:45.178932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.178110 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:08:45.179368 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:45.179279 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b3c864_3667_4eb6_bbd1_4887545c1493.slice/crio-c3f870d6b5824b42a4b1f8ac9ab629f8ac511b7823480e3d3769a1b6fefa221b WatchSource:0}: Error finding container c3f870d6b5824b42a4b1f8ac9ab629f8ac511b7823480e3d3769a1b6fefa221b: Status 404 returned error can't find the container with id c3f870d6b5824b42a4b1f8ac9ab629f8ac511b7823480e3d3769a1b6fefa221b Oct 30 01:08:45.336539 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.336510 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:08:45.349400 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:08:45.349364 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11da6415_854c_4b21_b4da_98c83edddd39.slice/crio-40252fe3a9cd8092829c482e314341fa9624d1d2708f56ff9207f941db769ded WatchSource:0}: Error finding container 40252fe3a9cd8092829c482e314341fa9624d1d2708f56ff9207f941db769ded: Status 404 returned error can't find the container with id 40252fe3a9cd8092829c482e314341fa9624d1d2708f56ff9207f941db769ded Oct 30 01:08:45.699834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.699796 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" event={"ID":"11da6415-854c-4b21-b4da-98c83edddd39","Type":"ContainerStarted","Data":"40252fe3a9cd8092829c482e314341fa9624d1d2708f56ff9207f941db769ded"} Oct 30 01:08:45.701927 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.701895 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" event={"ID":"c7b3c864-3667-4eb6-bbd1-4887545c1493","Type":"ContainerStarted","Data":"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6"} Oct 30 01:08:45.702073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.701942 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" event={"ID":"c7b3c864-3667-4eb6-bbd1-4887545c1493","Type":"ContainerStarted","Data":"c3f870d6b5824b42a4b1f8ac9ab629f8ac511b7823480e3d3769a1b6fefa221b"} Oct 30 01:08:45.735826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.735778 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" podStartSLOduration=1.7357668259999999 podStartE2EDuration="1.735766826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:45.735367085 +0000 UTC m=+1516.577927677" watchObservedRunningTime="2026-10-30 01:08:45.735766826 +0000 UTC m=+1516.578327420" Oct 30 01:08:45.766790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:45.766768 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:46.711587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:46.711543 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" event={"ID":"11da6415-854c-4b21-b4da-98c83edddd39","Type":"ContainerStarted","Data":"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008"} Oct 30 01:08:46.712106 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:46.711833 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:46.748173 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:46.748122 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" podStartSLOduration=3.361626787 podStartE2EDuration="3.748109132s" totalImagesPullingTime="386.482345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:46.745450039 +0000 UTC m=+1517.588010631" watchObservedRunningTime="2026-10-30 01:08:46.748109132 +0000 UTC m=+1517.590669725" Oct 30 01:08:52.728151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:52.728124 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:08:54.133405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.133373 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:08:54.133959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.133607 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" podUID="891c5644-9720-4ea8-b01c-7bb81f26fefd" containerName="maas-api" containerID="cri-o://c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b" gracePeriod=30 Oct 30 01:08:54.567885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.563032 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:08:54.593820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.592899 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\") pod \"891c5644-9720-4ea8-b01c-7bb81f26fefd\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " Oct 30 01:08:54.593820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.593064 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\") pod \"891c5644-9720-4ea8-b01c-7bb81f26fefd\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " Oct 30 01:08:54.593820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.593182 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\") pod \"891c5644-9720-4ea8-b01c-7bb81f26fefd\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " Oct 30 01:08:54.593820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.593246 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\" (UniqueName: \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\") pod \"891c5644-9720-4ea8-b01c-7bb81f26fefd\" (UID: \"891c5644-9720-4ea8-b01c-7bb81f26fefd\") " Oct 30 01:08:54.594199 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.593991 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca" pod "891c5644-9720-4ea8-b01c-7bb81f26fefd" (UID: "891c5644-9720-4ea8-b01c-7bb81f26fefd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:54.594604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.594568 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/891c5644-9720-4ea8-b01c-7bb81f26fefd-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:54.613932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.603732 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs" pod "891c5644-9720-4ea8-b01c-7bb81f26fefd" (UID: "891c5644-9720-4ea8-b01c-7bb81f26fefd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:54.613932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.608809 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls" pod "891c5644-9720-4ea8-b01c-7bb81f26fefd" (UID: "891c5644-9720-4ea8-b01c-7bb81f26fefd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:54.613932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.609069 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd" pod "891c5644-9720-4ea8-b01c-7bb81f26fefd" (UID: "891c5644-9720-4ea8-b01c-7bb81f26fefd"). InnerVolumeSpecName "kube-api-access-lmlhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:54.698014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.697981 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:54.698014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.698012 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/891c5644-9720-4ea8-b01c-7bb81f26fefd-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:54.698014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.698024 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlhd\" (UniqueName: \"kubernetes.io/projected/891c5644-9720-4ea8-b01c-7bb81f26fefd-kube-api-access-lmlhd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:08:54.766357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.766272 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:54.778311 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.778273 5548 generic.go:356] "Generic (PLEG): container finished" podID="891c5644-9720-4ea8-b01c-7bb81f26fefd" containerID="c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b" exitCode=0 Oct 30 01:08:54.778547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.778524 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" event={"ID":"891c5644-9720-4ea8-b01c-7bb81f26fefd","Type":"ContainerDied","Data":"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b"} Oct 30 01:08:54.778626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.778582 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" event={"ID":"891c5644-9720-4ea8-b01c-7bb81f26fefd","Type":"ContainerDied","Data":"15992ad51a8a28cf201a3cf539026f4450452467074ba18f8a9afa250b09359a"} Oct 30 01:08:54.778626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.778612 5548 scope.go:122] "RemoveContainer" containerID="c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b" Oct 30 01:08:54.778744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.778733 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw" Oct 30 01:08:54.787536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.787516 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:08:54.799819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.799794 5548 scope.go:122] "RemoveContainer" containerID="c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b" Oct 30 01:08:54.800477 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:54.800451 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b\": container with ID starting with c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b not found: ID does not exist" containerID="c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b" Oct 30 01:08:54.800565 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.800488 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b"} err="failed to get container status \"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b\": rpc error: code = NotFound desc = could not find container \"c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b\": container with ID starting with c256809561de22c252725bd612afaf52081caa45c5154ba55c41c9fa8363634b not found: ID does not exist" Oct 30 01:08:54.887560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.886976 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:08:54.937288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:54.937248 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-jkxfw"] Oct 30 01:08:55.268128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:55.268098 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891c5644-9720-4ea8-b01c-7bb81f26fefd" path="/var/lib/kubelet/pods/891c5644-9720-4ea8-b01c-7bb81f26fefd/volumes" Oct 30 01:08:55.430803 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:08:55.430766 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:08:55.777600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:08:55.777567 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:09:07.551492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.551114 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:09:07.552148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.551960 5548 cpu_manager.go:414] "containerMap: removing container" podUID="891c5644-9720-4ea8-b01c-7bb81f26fefd" containerName="maas-api" Oct 30 01:09:07.552148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.551976 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891c5644-9720-4ea8-b01c-7bb81f26fefd" containerName="maas-api" Oct 30 01:09:07.553981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.553940 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="891c5644-9720-4ea8-b01c-7bb81f26fefd" containerName="maas-api" Oct 30 01:09:07.560345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.560317 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.618340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.618312 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e233f\"/\"openshift-service-ca.crt\"" Oct 30 01:09:07.622973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.618431 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e233f\"/\"test-model-7e233f-kserve-self-signed-certs\"" Oct 30 01:09:07.622973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.618495 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e233f\"/\"default-dockercfg-w5xz8\"" Oct 30 01:09:07.622973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.618579 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e233f\"/\"kube-root-ca.crt\"" Oct 30 01:09:07.623512 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.623478 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:09:07.655326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655280 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.655517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655400 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.655517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655428 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqc5h\" (UniqueName: \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.655517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655468 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.655517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655503 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.655734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.655642 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757072 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757173 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757201 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqc5h\" (UniqueName: \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757238 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757272 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757444 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.757880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.757832 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.758078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.758056 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.758204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.758188 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.760291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.760265 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.760796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.760775 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.804659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.804581 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqc5h\" (UniqueName: \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\") pod \"test-model-7e233f-kserve-55d79f5d9f-kb4kq\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:07.924885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:07.924104 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:08.398744 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:09:08.398708 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcaba65_6741_4301_9028_4e605460cec7.slice/crio-02c35072756b8e631bbb014a8572ed09bb3ddc72ddab41fddf6728c685673177 WatchSource:0}: Error finding container 02c35072756b8e631bbb014a8572ed09bb3ddc72ddab41fddf6728c685673177: Status 404 returned error can't find the container with id 02c35072756b8e631bbb014a8572ed09bb3ddc72ddab41fddf6728c685673177 Oct 30 01:09:08.399296 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:08.399273 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:09:08.915180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:08.915143 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" event={"ID":"6dcaba65-6741-4301-9028-4e605460cec7","Type":"ContainerStarted","Data":"6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486"} Oct 30 01:09:08.915180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:08.915186 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" event={"ID":"6dcaba65-6741-4301-9028-4e605460cec7","Type":"ContainerStarted","Data":"02c35072756b8e631bbb014a8572ed09bb3ddc72ddab41fddf6728c685673177"} Oct 30 01:09:08.925463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:08.925437 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:08.953338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:08.953288 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" podStartSLOduration=1.9532733389999999 podStartE2EDuration="1.953273339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:08.950791658 +0000 UTC m=+1539.793352263" watchObservedRunningTime="2026-10-30 01:09:08.953273339 +0000 UTC m=+1539.795833933" Oct 30 01:09:12.947818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.947785 5548 generic.go:356] "Generic (PLEG): container finished" podID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerID="939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0" exitCode=0 Oct 30 01:09:12.948273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.947871 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerDied","Data":"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0"} Oct 30 01:09:12.948323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.948313 5548 scope.go:122] "RemoveContainer" containerID="939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0" Oct 30 01:09:12.950098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.950072 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:12.950153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.950138 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:12.950188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:12.950172 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:13.957733 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:13.957696 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerStarted","Data":"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a"} Oct 30 01:09:14.311029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:14.310653 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:09:14.311029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:14.310970 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:14.311029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:14.311019 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:15.311418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:15.311381 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:15.311962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:15.311442 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:16.311518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:16.311482 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:16.311981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:16.311536 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:17.311395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:17.311357 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:17.311598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:17.311414 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:17.925227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:17.925194 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:17.938605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:17.938571 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:18.024309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:18.024278 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:09:18.310918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:18.310838 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:18.311073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:18.310922 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:19.311271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:19.311231 5548 patch_prober.go:29] interesting pod/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:09:19.311675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:19.311298 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 01:09:20.586333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:20.586305 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:09:20.586818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:20.586807 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:09:20.587759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:20.587740 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:09:22.761690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:22.761661 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 01:09:22.762232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:22.761948 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" containerID="cri-o://9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121" gracePeriod=30 Oct 30 01:09:25.428409 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:25.428373 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:09:26.083886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.083829 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:09:26.084204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.084082 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" podUID="c5daaafb-a3e0-4f02-9409-9df2179a7671" containerName="maas-api" containerID="cri-o://aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a" gracePeriod=30 Oct 30 01:09:26.123275 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:26.123240 5548 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5daaafb_a3e0_4f02_9409_9df2179a7671.slice/crio-conmon-aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:09:26.123414 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:26.123292 5548 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5daaafb_a3e0_4f02_9409_9df2179a7671.slice/crio-conmon-aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:09:26.584531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.584501 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:09:26.721201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.721113 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\") pod \"c5daaafb-a3e0-4f02-9409-9df2179a7671\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " Oct 30 01:09:26.721201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.721160 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\") pod \"c5daaafb-a3e0-4f02-9409-9df2179a7671\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " Oct 30 01:09:26.721419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.721334 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\") pod \"c5daaafb-a3e0-4f02-9409-9df2179a7671\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " Oct 30 01:09:26.721675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.721542 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\") pod \"c5daaafb-a3e0-4f02-9409-9df2179a7671\" (UID: \"c5daaafb-a3e0-4f02-9409-9df2179a7671\") " Oct 30 01:09:26.722824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.722791 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca" pod "c5daaafb-a3e0-4f02-9409-9df2179a7671" (UID: "c5daaafb-a3e0-4f02-9409-9df2179a7671"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:26.727584 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.727542 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls" pod "c5daaafb-a3e0-4f02-9409-9df2179a7671" (UID: "c5daaafb-a3e0-4f02-9409-9df2179a7671"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:26.736602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.736571 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs" pod "c5daaafb-a3e0-4f02-9409-9df2179a7671" (UID: "c5daaafb-a3e0-4f02-9409-9df2179a7671"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:09:26.737109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.737091 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd" pod "c5daaafb-a3e0-4f02-9409-9df2179a7671" (UID: "c5daaafb-a3e0-4f02-9409-9df2179a7671"). InnerVolumeSpecName "kube-api-access-65qnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:26.824093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.824063 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:26.824093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.824093 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5daaafb-a3e0-4f02-9409-9df2179a7671-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:26.824331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.824111 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qnd\" (UniqueName: \"kubernetes.io/projected/c5daaafb-a3e0-4f02-9409-9df2179a7671-kube-api-access-65qnd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:26.824331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:26.824124 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5daaafb-a3e0-4f02-9409-9df2179a7671-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:27.077380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.077285 5548 generic.go:356] "Generic (PLEG): container finished" podID="c5daaafb-a3e0-4f02-9409-9df2179a7671" containerID="aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a" exitCode=0 Oct 30 01:09:27.077380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.077357 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" event={"ID":"c5daaafb-a3e0-4f02-9409-9df2179a7671","Type":"ContainerDied","Data":"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a"} Oct 30 01:09:27.077380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.077375 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" Oct 30 01:09:27.077655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.077425 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq" event={"ID":"c5daaafb-a3e0-4f02-9409-9df2179a7671","Type":"ContainerDied","Data":"dfd0575b666c0601aaa465fc6a03c4e4a66c1ec7916262e55043bf6a2ba58542"} Oct 30 01:09:27.077655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.077448 5548 scope.go:122] "RemoveContainer" containerID="aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a" Oct 30 01:09:27.088231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.088013 5548 scope.go:122] "RemoveContainer" containerID="aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a" Oct 30 01:09:27.088335 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:27.088316 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a\": container with ID starting with aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a not found: ID does not exist" containerID="aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a" Oct 30 01:09:27.088377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.088343 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a"} err="failed to get container status \"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a\": rpc error: code = NotFound desc = could not find container \"aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a\": container with ID starting with aec588ea1f9cb73c95ebd0a7dc33c64c50cce6d54fb52107b05bbbd8b091f96a not found: ID does not exist" Oct 30 01:09:27.133147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.133118 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:09:27.144681 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.144652 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-288da6-57fb9d7f78-gflsq"] Oct 30 01:09:27.272108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:27.272068 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5daaafb-a3e0-4f02-9409-9df2179a7671" path="/var/lib/kubelet/pods/c5daaafb-a3e0-4f02-9409-9df2179a7671/volumes" Oct 30 01:09:28.048215 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.048151 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 01:09:28.085692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.085658 5548 generic.go:356] "Generic (PLEG): container finished" podID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerID="9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121" exitCode=0 Oct 30 01:09:28.085848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.085706 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" Oct 30 01:09:28.085848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.085745 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" event={"ID":"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4","Type":"ContainerDied","Data":"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121"} Oct 30 01:09:28.085848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.085791 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9" event={"ID":"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4","Type":"ContainerDied","Data":"dd484ae6c3a3b49daf579c46f1f9cbeb7633fe774a2b3094ad94895eb4c8f0dd"} Oct 30 01:09:28.085848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.085813 5548 scope.go:122] "RemoveContainer" containerID="9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121" Oct 30 01:09:28.096165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.095965 5548 scope.go:122] "RemoveContainer" containerID="9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121" Oct 30 01:09:28.096262 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:28.096236 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121\": container with ID starting with 9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121 not found: ID does not exist" containerID="9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121" Oct 30 01:09:28.096307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.096269 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121"} err="failed to get container status \"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121\": rpc error: code = NotFound desc = could not find container \"9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121\": container with ID starting with 9c5a69cbbe5dac1a52fb5e854f2f8aef1c8a320941ee3986bcf36f4aa51fc121 not found: ID does not exist" Oct 30 01:09:28.147445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147409 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147504 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147546 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147601 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147643 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147676 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.147815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147754 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:28.147815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147776 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.148082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147876 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.148082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.147930 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\") pod \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\" (UID: \"ab0d01bd-dd62-4ac8-a647-bd8214cc15e4\") " Oct 30 01:09:28.148082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:28.148247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148208 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:28.148797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148620 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.148797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148644 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.148797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148657 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.148797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.148757 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:28.149247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.149223 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:28.150676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.150655 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "kube-api-access-bn48c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:28.150775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.150689 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:28.151202 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.151179 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:28.151405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.151379 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo" pod "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" (UID: "ab0d01bd-dd62-4ac8-a647-bd8214cc15e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:28.250535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250506 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn48c\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-kube-api-access-bn48c\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.250535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250534 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.250535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250543 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.250784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250553 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.250784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250561 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.250784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.250568 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:28.431158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.431128 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 01:09:28.434690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:28.434666 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-n25b9"] Oct 30 01:09:29.266017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:29.265988 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" path="/var/lib/kubelet/pods/ab0d01bd-dd62-4ac8-a647-bd8214cc15e4/volumes" Oct 30 01:09:34.937964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:34.937928 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:09:34.938575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:34.938174 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" containerID="cri-o://781406937bc9ba52c0a95e7f5c280b81b54f086a3429b226c07dae0d9b172f9f" gracePeriod=30 Oct 30 01:09:36.263625 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.263593 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:09:36.265131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265103 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c5daaafb-a3e0-4f02-9409-9df2179a7671" containerName="maas-api" Oct 30 01:09:36.265283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265133 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5daaafb-a3e0-4f02-9409-9df2179a7671" containerName="maas-api" Oct 30 01:09:36.265283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265187 5548 cpu_manager.go:414] "containerMap: removing container" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" Oct 30 01:09:36.265283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265199 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" Oct 30 01:09:36.265488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265364 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5daaafb-a3e0-4f02-9409-9df2179a7671" containerName="maas-api" Oct 30 01:09:36.265488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.265378 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0d01bd-dd62-4ac8-a647-bd8214cc15e4" containerName="istio-proxy" Oct 30 01:09:36.269386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.269366 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.285161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.285134 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52f1cb-openshift-default-dockercfg-bbr78\"" Oct 30 01:09:36.301941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.301908 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:09:36.365822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.365781 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxdw\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.365893 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366074 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366099 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366230 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366129 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366230 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366205 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366261 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366446 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.366527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.366502 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.468613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.468562 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxdw\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.469455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.469342 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.477085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.477055 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.477638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.477620 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.478266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.478246 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.478435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.478418 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.478606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.478591 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.488892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.478738 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.489696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.489674 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.491846 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.470133 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.492036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.477555 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.492208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.490713 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.492331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.491358 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.492432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.491600 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.508841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.508545 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxdw\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.509699 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.509675 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.517940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.517818 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.541884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.535378 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\") pod \"e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:36.593888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:36.591240 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:37.021890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.017931 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:09:37.031724 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:09:37.031693 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746e4137_cbc7_44a4_a0c8_d22ab1ca2cfd.slice/crio-ba21097d4172938c3854e4285ade3977d399019744e58febb4aa4bcb6845bc70 WatchSource:0}: Error finding container ba21097d4172938c3854e4285ade3977d399019744e58febb4aa4bcb6845bc70: Status 404 returned error can't find the container with id ba21097d4172938c3854e4285ade3977d399019744e58febb4aa4bcb6845bc70 Oct 30 01:09:37.035270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.035225 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:37.035377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.035329 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:37.035419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.035379 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:37.179958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.179819 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerStarted","Data":"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f"} Oct 30 01:09:37.179958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.179932 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerStarted","Data":"ba21097d4172938c3854e4285ade3977d399019744e58febb4aa4bcb6845bc70"} Oct 30 01:09:37.230643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.230587 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podStartSLOduration=1.230568634 podStartE2EDuration="1.230568634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:37.2262773 +0000 UTC m=+1568.068837895" watchObservedRunningTime="2026-10-30 01:09:37.230568634 +0000 UTC m=+1568.073129227" Oct 30 01:09:37.591912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:37.591879 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:38.134676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:38.134643 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:09:38.134937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:38.134884 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" containerID="cri-o://fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c" gracePeriod=30 Oct 30 01:09:38.672888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:38.667180 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:39.201753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:39.201718 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:39.202757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:39.202739 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:09:40.211195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.211116 5548 generic.go:356] "Generic (PLEG): container finished" podID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerID="781406937bc9ba52c0a95e7f5c280b81b54f086a3429b226c07dae0d9b172f9f" exitCode=0 Oct 30 01:09:40.211670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.211189 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerDied","Data":"781406937bc9ba52c0a95e7f5c280b81b54f086a3429b226c07dae0d9b172f9f"} Oct 30 01:09:40.211670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.211253 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" event={"ID":"33a03d5e-3270-4c4a-946e-54a9a8a348eb","Type":"ContainerDied","Data":"554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8"} Oct 30 01:09:40.211670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.211276 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554ae52ca3cb0d37a770fcfd3a4eccae99c0b22ddf37edd664a21adfd79a19a8" Oct 30 01:09:40.211670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.211295 5548 scope.go:122] "RemoveContainer" containerID="67d5dcdb1962a743d7aa7f2225c50135a4528b37a1a78dfa12379e9050a4b317" Oct 30 01:09:40.238287 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.238261 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.355777 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.355848 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356062 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356198 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356287 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356346 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356472 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356502 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.356687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.356530 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\") pod \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\" (UID: \"33a03d5e-3270-4c4a-946e-54a9a8a348eb\") " Oct 30 01:09:40.357473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.357434 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:40.357607 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.357486 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.358471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.358214 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.358471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.358307 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.358471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.358451 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.359465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.359371 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.359465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.359402 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.359465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.359415 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.359465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.359428 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.359465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.359441 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.360272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.360221 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:40.360272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.360247 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:40.363900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.361968 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "kube-api-access-cqvb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:40.363900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.362102 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy" pod "33a03d5e-3270-4c4a-946e-54a9a8a348eb" (UID: "33a03d5e-3270-4c4a-946e-54a9a8a348eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:40.460388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.460351 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.460388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.460379 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.460388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.460392 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvb2\" (UniqueName: \"kubernetes.io/projected/33a03d5e-3270-4c4a-946e-54a9a8a348eb-kube-api-access-cqvb2\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:40.460620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:40.460404 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a03d5e-3270-4c4a-946e-54a9a8a348eb-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:41.219797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:41.219768 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj" Oct 30 01:09:41.289988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:41.289959 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:09:41.296103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:41.296079 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-288da6-openshift-default-68599494b9-5kwfj"] Oct 30 01:09:43.265729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.265697 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" path="/var/lib/kubelet/pods/33a03d5e-3270-4c4a-946e-54a9a8a348eb/volumes" Oct 30 01:09:43.514995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.513286 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:09:43.647370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647337 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647391 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647483 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647502 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647546 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647571 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647622 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647662 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.647706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647694 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:43.648032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647770 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\") pod \"9dda9ec9-f322-44fe-b855-83f52ed8f260\" (UID: \"9dda9ec9-f322-44fe-b855-83f52ed8f260\") " Oct 30 01:09:43.648032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.647966 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:43.648152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648072 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:43.648152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648089 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:43.648289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648248 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:43.648710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648672 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.648710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648698 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.648908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648712 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dda9ec9-f322-44fe-b855-83f52ed8f260-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.648908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648726 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.648908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.648739 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.650137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.650114 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:43.650137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.650128 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:43.650268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.650129 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "kube-api-access-8hjht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:43.650407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.650393 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo" pod "9dda9ec9-f322-44fe-b855-83f52ed8f260" (UID: "9dda9ec9-f322-44fe-b855-83f52ed8f260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:43.749845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.749761 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.749845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.749799 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjht\" (UniqueName: \"kubernetes.io/projected/9dda9ec9-f322-44fe-b855-83f52ed8f260-kube-api-access-8hjht\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.749845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.749810 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.749845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:43.749819 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dda9ec9-f322-44fe-b855-83f52ed8f260-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:09:44.242548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.242516 5548 generic.go:356] "Generic (PLEG): container finished" podID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerID="fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c" exitCode=0 Oct 30 01:09:44.242749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.242566 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" Oct 30 01:09:44.242749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.242594 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" event={"ID":"9dda9ec9-f322-44fe-b855-83f52ed8f260","Type":"ContainerDied","Data":"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c"} Oct 30 01:09:44.242749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.242650 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c" event={"ID":"9dda9ec9-f322-44fe-b855-83f52ed8f260","Type":"ContainerDied","Data":"87d1013eaa0b89738eb5db05bf3e74aa9657935fc817133efd0973d89b38460c"} Oct 30 01:09:44.242749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.242675 5548 scope.go:122] "RemoveContainer" containerID="fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c" Oct 30 01:09:44.253731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.253575 5548 scope.go:122] "RemoveContainer" containerID="fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c" Oct 30 01:09:44.253904 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:44.253846 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c\": container with ID starting with fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c not found: ID does not exist" containerID="fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c" Oct 30 01:09:44.253958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.253921 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c"} err="failed to get container status \"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c\": rpc error: code = NotFound desc = could not find container \"fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c\": container with ID starting with fc677b8ba5f6ea5e8a62d09abb1ba57d832dda68798a92deeaee032d32ccaa8c not found: ID does not exist" Oct 30 01:09:44.303795 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.303762 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:09:44.314326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:44.314292 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-xkq4c"] Oct 30 01:09:45.265921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:45.265887 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" path="/var/lib/kubelet/pods/9dda9ec9-f322-44fe-b855-83f52ed8f260/volumes" Oct 30 01:09:47.166138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.166107 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:09:47.167831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.167803 5548 cpu_manager.go:414] "containerMap: removing container" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" Oct 30 01:09:47.168023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.168008 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" Oct 30 01:09:47.168151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.168141 5548 cpu_manager.go:414] "containerMap: removing container" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.168227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.168217 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.179036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.179015 5548 cpu_manager.go:414] "containerMap: removing container" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.179242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.179227 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.179650 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.179622 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dda9ec9-f322-44fe-b855-83f52ed8f260" containerName="istio-proxy" Oct 30 01:09:47.179793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.179783 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.179934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.179923 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a03d5e-3270-4c4a-946e-54a9a8a348eb" containerName="istio-proxy" Oct 30 01:09:47.223518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.223468 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:09:47.223782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.223470 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.234954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.234780 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.234954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.234891 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.234956 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rhh\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235007 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235051 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235080 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235187 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235409 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.235488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.235457 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.237591 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.236955 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e92dab-openshift-default-dockercfg-9pm7p\"" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382062 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rhh\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382131 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382171 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382198 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382297 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382463 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382495 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382552 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.382882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.382637 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.393010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.383726 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.393010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.384576 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.393010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.384970 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.393010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.385211 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.393010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.386281 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.439896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.439800 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.440896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.440848 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.459902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.453632 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rhh\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.461758 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.461699 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\") pod \"e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:47.549175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:47.549137 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:48.055565 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.055427 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:09:48.056123 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:09:48.056095 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf382efd_8ef0_44fc_a504_b667e138ed43.slice/crio-da7658be6838514ccd27a1ab33d329a3d8dda06e119f6699b59461815aa95287 WatchSource:0}: Error finding container da7658be6838514ccd27a1ab33d329a3d8dda06e119f6699b59461815aa95287: Status 404 returned error can't find the container with id da7658be6838514ccd27a1ab33d329a3d8dda06e119f6699b59461815aa95287 Oct 30 01:09:48.059270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.059234 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:48.059331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.059320 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:48.059379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.059367 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:48.376944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.372926 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerStarted","Data":"1668f8e1d4fbc7f9591a800e25f332578b2db145a1b466bec5eac7d61219a13b"} Oct 30 01:09:48.376944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.372995 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerStarted","Data":"da7658be6838514ccd27a1ab33d329a3d8dda06e119f6699b59461815aa95287"} Oct 30 01:09:48.419132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.419071 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podStartSLOduration=1.419050018 podStartE2EDuration="1.419050018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:48.413356977 +0000 UTC m=+1579.255917590" watchObservedRunningTime="2026-10-30 01:09:48.419050018 +0000 UTC m=+1579.261610612" Oct 30 01:09:48.550355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:48.550311 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:49.556214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:49.556180 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:50.399469 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:50.399436 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:50.400387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:50.400368 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:09:55.429046 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:09:55.428995 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:09:56.443797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.443762 5548 generic.go:356] "Generic (PLEG): container finished" podID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerID="89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca" exitCode=0 Oct 30 01:09:56.444238 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.443826 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerDied","Data":"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca"} Oct 30 01:09:56.444324 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.444307 5548 scope.go:122] "RemoveContainer" containerID="89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca" Oct 30 01:09:56.446175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.446147 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:56.446254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.446213 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:56.446254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:56.446246 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:09:57.452899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:57.452851 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerStarted","Data":"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836"} Oct 30 01:09:57.500505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:57.500471 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:09:57.500695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:57.500656 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:09:57.500759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:57.500697 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:09:58.500782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:58.500748 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:09:58.501199 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:58.500812 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:09:59.511883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:59.511036 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:09:59.511883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:09:59.511099 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:10:00.500413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:00.500329 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:10:00.500413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:00.500397 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:10:01.499950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:01.499898 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:10:01.500399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:01.499954 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:10:02.499963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:02.499931 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:10:02.500399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:02.499982 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 01:10:02.597515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:02.597485 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:10:03.505103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:03.505071 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:10:03.525848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:03.525819 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:10:07.385688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.385653 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:10:07.389114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.389093 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.402126 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.402104 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7e233f-openshift-default-dockercfg-b8njx\"" Oct 30 01:10:07.409876 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.409834 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.409989 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410081 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410157 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410293 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410246 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410363 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410288 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410368 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410410 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.410525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.410505 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdmp\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.415997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.415963 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:10:07.511895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.511832 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512066 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.511933 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512066 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.511991 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512066 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512037 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512083 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512148 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdmp\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512272 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512285 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512404 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512376 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512432 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512373 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512587 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512732 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.512790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.512760 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.514606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.514581 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.514940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.514922 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.529965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.529942 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.530085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.530068 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdmp\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\") pod \"e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:07.702326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:07.702296 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:08.022637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.022614 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:10:08.023099 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:10:08.023070 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29859192_79d3_458a_8bd5_0dea435986b2.slice/crio-67bd35158b52adb8dfad858f201ab34628fcd092aba2f2461c1ed2130551563a WatchSource:0}: Error finding container 67bd35158b52adb8dfad858f201ab34628fcd092aba2f2461c1ed2130551563a: Status 404 returned error can't find the container with id 67bd35158b52adb8dfad858f201ab34628fcd092aba2f2461c1ed2130551563a Oct 30 01:10:08.026249 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.026210 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:08.026346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.026290 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:08.026346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.026343 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:08.577400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.575964 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" event={"ID":"29859192-79d3-458a-8bd5-0dea435986b2","Type":"ContainerStarted","Data":"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d"} Oct 30 01:10:08.577400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.576046 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" event={"ID":"29859192-79d3-458a-8bd5-0dea435986b2","Type":"ContainerStarted","Data":"67bd35158b52adb8dfad858f201ab34628fcd092aba2f2461c1ed2130551563a"} Oct 30 01:10:08.626170 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.626014 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podStartSLOduration=1.6259953299999998 podStartE2EDuration="1.62599533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:08.624406917 +0000 UTC m=+1599.466967510" watchObservedRunningTime="2026-10-30 01:10:08.62599533 +0000 UTC m=+1599.468555924" Oct 30 01:10:08.703595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:08.703560 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:09.705083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:09.705046 5548 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:10:09.705496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:09.705109 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 01:10:10.703480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:10.703446 5548 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:10:10.703674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:10.703510 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 01:10:11.703660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:11.703619 5548 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:10:11.704073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:11.703683 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 01:10:12.703472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:12.703432 5548 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:10:12.703683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:12.703485 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 01:10:13.703723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.703691 5548 patch_prober.go:29] interesting pod/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:10:13.704274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.703764 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 01:10:13.907593 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.907561 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:10:13.911240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.911217 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:13.925075 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.925047 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52f1cb-serving-cert\"" Oct 30 01:10:13.938592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:13.938559 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:10:14.025673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.025582 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.025673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.025662 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.025946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.025824 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4xm\" (UniqueName: \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.026001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.025950 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.126991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.126960 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.127172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.127070 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4xm\" (UniqueName: \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.127172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.127130 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.127360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.127344 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.127596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.127573 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.129901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.129846 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.130027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.130003 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.164707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.164676 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4xm\" (UniqueName: \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\") pod \"maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.225573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.225546 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:14.575578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.575553 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:10:14.576347 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:10:14.576317 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53391acd_093d_424a_9829_af156e13d521.slice/crio-03766461b5bd857b74c0beea33ab529a35c4f53dceaf09524c48a9a302b54c81 WatchSource:0}: Error finding container 03766461b5bd857b74c0beea33ab529a35c4f53dceaf09524c48a9a302b54c81: Status 404 returned error can't find the container with id 03766461b5bd857b74c0beea33ab529a35c4f53dceaf09524c48a9a302b54c81 Oct 30 01:10:14.636716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.636677 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" event={"ID":"53391acd-093d-424a-9829-af156e13d521","Type":"ContainerStarted","Data":"03766461b5bd857b74c0beea33ab529a35c4f53dceaf09524c48a9a302b54c81"} Oct 30 01:10:14.707747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.707707 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:14.708312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.708292 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:14.711170 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:14.710470 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:10:15.644981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:15.644949 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" event={"ID":"53391acd-093d-424a-9829-af156e13d521","Type":"ContainerStarted","Data":"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f"} Oct 30 01:10:15.645143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:15.645016 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:15.680395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:15.680350 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" podStartSLOduration=2.1832778409999998 podStartE2EDuration="2.680335904s" totalImagesPullingTime="497.058063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:15.679757649 +0000 UTC m=+1606.522318241" watchObservedRunningTime="2026-10-30 01:10:15.680335904 +0000 UTC m=+1606.522896497" Oct 30 01:10:21.659817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:21.659780 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:10:25.428333 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:10:25.428296 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:10:51.442064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:51.442022 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:10:51.442660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:51.442098 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:10:54.111325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.111296 5548 generic.go:356] "Generic (PLEG): container finished" podID="af382efd-8ef0-44fc-a504-b667e138ed43" containerID="1668f8e1d4fbc7f9591a800e25f332578b2db145a1b466bec5eac7d61219a13b" exitCode=0 Oct 30 01:10:54.111827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.111371 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerDied","Data":"1668f8e1d4fbc7f9591a800e25f332578b2db145a1b466bec5eac7d61219a13b"} Oct 30 01:10:54.111827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.111807 5548 scope.go:122] "RemoveContainer" containerID="1668f8e1d4fbc7f9591a800e25f332578b2db145a1b466bec5eac7d61219a13b" Oct 30 01:10:54.113380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113354 5548 generic.go:356] "Generic (PLEG): container finished" podID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerID="2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f" exitCode=0 Oct 30 01:10:54.113494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113435 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerDied","Data":"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f"} Oct 30 01:10:54.113728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113704 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.113775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113766 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.113817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113798 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.113913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.113797 5548 scope.go:122] "RemoveContainer" containerID="2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f" Oct 30 01:10:54.115413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.115387 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.115492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.115448 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.115492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.115481 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:10:54.202461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:54.202432 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:10:55.125506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.125471 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerStarted","Data":"2c7c997d5c8b0a23fc0682c2fdabc359aa26dcc94c44dd03cb632466dcae338a"} Oct 30 01:10:55.127712 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.127683 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerStarted","Data":"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1"} Oct 30 01:10:55.447832 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:10:55.447800 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:10:55.550814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.550787 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:10:55.551040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.551020 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:10:55.551091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.551066 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:10:55.594543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.592024 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:10:55.615888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.597003 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:10:55.615888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:55.597079 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:10:56.549955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:56.549925 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:10:56.550451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:56.550424 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:10:56.593885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:56.593219 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:10:56.593885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:56.593289 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:10:57.550418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:57.550387 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:10:57.550820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:57.550463 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:10:57.593886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:57.593318 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:10:57.593886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:57.593393 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:10:58.550491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:58.550453 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:10:58.550948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:58.550514 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:10:58.616887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:58.603263 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:10:58.616887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:58.603339 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:10:59.550301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:59.550263 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:10:59.550516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:59.550323 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:10:59.597051 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:59.597017 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:10:59.597337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:10:59.597310 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:11:00.550255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:00.550220 5548 patch_prober.go:29] interesting pod/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:11:00.550442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:00.550284 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 01:11:00.592988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:00.592955 5548 patch_prober.go:29] interesting pod/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:11:00.593385 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:00.593010 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 01:11:01.554489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.554459 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:11:01.554997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.554975 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:11:01.555759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.555742 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:11:01.599349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.599321 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:11:01.599819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.599792 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:11:01.600710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:01.600681 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:11:20.396528 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.396494 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:11:20.401685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.401651 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.427193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.427137 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:11:20.432919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.430963 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-29145c-openshift-default-dockercfg-5nhvg\"" Oct 30 01:11:20.487310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.487278 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:11:20.519963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.519931 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.520217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.520198 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.520361 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.520347 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.520567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.520552 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.520908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.520889 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.521067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.521053 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.521214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.521197 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rndr\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.521341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.521327 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.521465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.521452 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.629699 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.629666 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.629914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.629720 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.629914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.629872 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.629921 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.629965 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.630090 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.630123 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.630187 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.630274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.630243 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rndr\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.631225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.631199 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.632339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.632321 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.640892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.632585 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.640892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.632848 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.641597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.641575 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.641977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.641956 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.662551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.662469 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rndr\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.662978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.662953 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.681390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.675967 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:20.724409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:20.723681 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:21.236538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:21.236437 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:11:21.361164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:21.360349 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:11:21.361164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:21.360442 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:11:21.361164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:21.360486 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:11:22.368384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:22.368349 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerStarted","Data":"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453"} Oct 30 01:11:22.368766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:22.368404 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerStarted","Data":"5c3350177d3121d8abdfe7cfa5b6d54321c2c6e394601a3c01b1f30fe058ff80"} Oct 30 01:11:22.428168 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:22.428103 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podStartSLOduration=2.428079182 podStartE2EDuration="2.428079182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:22.413572547 +0000 UTC m=+1673.256133142" watchObservedRunningTime="2026-10-30 01:11:22.428079182 +0000 UTC m=+1673.270639806" Oct 30 01:11:22.729891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:22.727170 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:22.746438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:22.746409 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:23.375607 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:23.375578 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:23.376568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:23.376552 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:11:25.429670 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:25.429629 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:11:35.534804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.534716 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerName="main" containerID="cri-o://7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6" gracePeriod=45 Oct 30 01:11:35.829218 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.829195 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:11:35.930428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930402 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\" (UniqueName: \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930458 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930560 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930618 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930712 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930771 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\") pod \"c7b3c864-3667-4eb6-bbd1-4887545c1493\" (UID: \"c7b3c864-3667-4eb6-bbd1-4887545c1493\") " Oct 30 01:11:35.930835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930799 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:35.930970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930844 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:35.930970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.930900 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:35.931356 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.931333 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:35.931469 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.931382 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:35.931469 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.931396 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:35.932942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.932915 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:35.933044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.933004 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "kube-api-access-56qdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:35.933044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:35.933024 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs" pod "c7b3c864-3667-4eb6-bbd1-4887545c1493" (UID: "c7b3c864-3667-4eb6-bbd1-4887545c1493"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:36.032475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.032446 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b3c864-3667-4eb6-bbd1-4887545c1493-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:36.032475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.032472 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b3c864-3667-4eb6-bbd1-4887545c1493-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:36.032475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.032481 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qdk\" (UniqueName: \"kubernetes.io/projected/c7b3c864-3667-4eb6-bbd1-4887545c1493-kube-api-access-56qdk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:36.485819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.485780 5548 generic.go:356] "Generic (PLEG): container finished" podID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerID="7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6" exitCode=0 Oct 30 01:11:36.486015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.485829 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" Oct 30 01:11:36.486015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.485891 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" event={"ID":"c7b3c864-3667-4eb6-bbd1-4887545c1493","Type":"ContainerDied","Data":"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6"} Oct 30 01:11:36.486015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.485937 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" event={"ID":"c7b3c864-3667-4eb6-bbd1-4887545c1493","Type":"ContainerDied","Data":"c3f870d6b5824b42a4b1f8ac9ab629f8ac511b7823480e3d3769a1b6fefa221b"} Oct 30 01:11:36.486015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.485954 5548 scope.go:122] "RemoveContainer" containerID="7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6" Oct 30 01:11:36.497848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.497826 5548 scope.go:122] "RemoveContainer" containerID="7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6" Oct 30 01:11:36.498178 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:36.498158 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6\": container with ID starting with 7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6 not found: ID does not exist" containerID="7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6" Oct 30 01:11:36.498274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.498223 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6"} err="failed to get container status \"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6\": rpc error: code = NotFound desc = could not find container \"7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6\": container with ID starting with 7b967595e01689a2374750f0bbfcde299898b91be2db2440422c7ebe58526aa6 not found: ID does not exist" Oct 30 01:11:36.531158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.531133 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:11:36.541213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.541190 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx"] Oct 30 01:11:36.766777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.766700 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a3204e/test-model-a3204e-kserve-96df9444f-gxbvx" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:11:36.892913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:36.892806 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:11:37.283255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:37.283215 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" path="/var/lib/kubelet/pods/c7b3c864-3667-4eb6-bbd1-4887545c1493/volumes" Oct 30 01:11:44.150076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.150042 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:11:44.150834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.150817 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerName="main" Oct 30 01:11:44.150948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.150846 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerName="main" Oct 30 01:11:44.151007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.150996 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b3c864-3667-4eb6-bbd1-4887545c1493" containerName="main" Oct 30 01:11:44.155688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.155673 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.172998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.172974 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:11:44.178486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.178461 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-687471-serving-cert\"" Oct 30 01:11:44.258407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.258372 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.258597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.258576 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.258664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.258650 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mjd\" (UniqueName: \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.258799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.258782 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.360402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.360366 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.360604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.360430 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mjd\" (UniqueName: \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.360604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.360532 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.360889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.360720 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.361197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.361179 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.363133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.363111 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.363277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.363247 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.378481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.378457 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mjd\" (UniqueName: \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\") pod \"maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.471318 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.471233 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:44.787192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.787164 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:11:44.796349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:44.796321 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:11:45.572121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.572083 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" event={"ID":"489949ee-01ce-436f-b1a9-c9e667e5e350","Type":"ContainerStarted","Data":"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32"} Oct 30 01:11:45.572588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.572136 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" event={"ID":"489949ee-01ce-436f-b1a9-c9e667e5e350","Type":"ContainerStarted","Data":"8b82ed6b281c70ffea2b36c22abf4d1d53b33f87644b56c5d81ea886ccdf74b0"} Oct 30 01:11:45.572588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.572301 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:45.630107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.630061 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" podStartSLOduration=1.274514965 podStartE2EDuration="1.63004594s" totalImagesPullingTime="355.530975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:45.607831933 +0000 UTC m=+1696.450392538" watchObservedRunningTime="2026-10-30 01:11:45.63004594 +0000 UTC m=+1696.472606533" Oct 30 01:11:45.633707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.633669 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:11:45.633952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:45.633923 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" podUID="11da6415-854c-4b21-b4da-98c83edddd39" containerName="maas-api" containerID="cri-o://35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008" gracePeriod=30 Oct 30 01:11:46.268850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.268824 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:11:46.376581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.376542 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\") pod \"11da6415-854c-4b21-b4da-98c83edddd39\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " Oct 30 01:11:46.376999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.376974 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") pod \"11da6415-854c-4b21-b4da-98c83edddd39\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " Oct 30 01:11:46.377304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.377290 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\") pod \"11da6415-854c-4b21-b4da-98c83edddd39\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " Oct 30 01:11:46.377415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.377403 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\" (UniqueName: \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\") pod \"11da6415-854c-4b21-b4da-98c83edddd39\" (UID: \"11da6415-854c-4b21-b4da-98c83edddd39\") " Oct 30 01:11:46.389743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.389121 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca" pod "11da6415-854c-4b21-b4da-98c83edddd39" (UID: "11da6415-854c-4b21-b4da-98c83edddd39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:46.394919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.393129 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls" pod "11da6415-854c-4b21-b4da-98c83edddd39" (UID: "11da6415-854c-4b21-b4da-98c83edddd39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:46.394919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.394758 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt" pod "11da6415-854c-4b21-b4da-98c83edddd39" (UID: "11da6415-854c-4b21-b4da-98c83edddd39"). InnerVolumeSpecName "kube-api-access-8vvkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:46.417150 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.417113 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs" pod "11da6415-854c-4b21-b4da-98c83edddd39" (UID: "11da6415-854c-4b21-b4da-98c83edddd39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:46.486605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.486534 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11da6415-854c-4b21-b4da-98c83edddd39-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:46.486605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.486563 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:46.486605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.486571 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11da6415-854c-4b21-b4da-98c83edddd39-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:46.486605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.486580 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvkt\" (UniqueName: \"kubernetes.io/projected/11da6415-854c-4b21-b4da-98c83edddd39-kube-api-access-8vvkt\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:46.580484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.580450 5548 generic.go:356] "Generic (PLEG): container finished" podID="11da6415-854c-4b21-b4da-98c83edddd39" containerID="35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008" exitCode=0 Oct 30 01:11:46.580976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.580502 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" Oct 30 01:11:46.580976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.580540 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" event={"ID":"11da6415-854c-4b21-b4da-98c83edddd39","Type":"ContainerDied","Data":"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008"} Oct 30 01:11:46.580976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.580601 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr" event={"ID":"11da6415-854c-4b21-b4da-98c83edddd39","Type":"ContainerDied","Data":"40252fe3a9cd8092829c482e314341fa9624d1d2708f56ff9207f941db769ded"} Oct 30 01:11:46.580976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.580622 5548 scope.go:122] "RemoveContainer" containerID="35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008" Oct 30 01:11:46.591834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.591815 5548 scope.go:122] "RemoveContainer" containerID="35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008" Oct 30 01:11:46.592207 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:46.592181 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008\": container with ID starting with 35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008 not found: ID does not exist" containerID="35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008" Oct 30 01:11:46.592304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.592228 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008"} err="failed to get container status \"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008\": rpc error: code = NotFound desc = could not find container \"35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008\": container with ID starting with 35b3e62ae4576bb3e74a92569ad7044c22778a51ecb4de1f516d824c509f0008 not found: ID does not exist" Oct 30 01:11:46.630976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.630948 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:11:46.640921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:46.640888 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da60de-6857bf985-4jwkr"] Oct 30 01:11:47.266500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:47.266468 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11da6415-854c-4b21-b4da-98c83edddd39" path="/var/lib/kubelet/pods/11da6415-854c-4b21-b4da-98c83edddd39/volumes" Oct 30 01:11:48.435645 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.435611 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:11:48.436591 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.436570 5548 cpu_manager.go:414] "containerMap: removing container" podUID="11da6415-854c-4b21-b4da-98c83edddd39" containerName="maas-api" Oct 30 01:11:48.436647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.436595 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11da6415-854c-4b21-b4da-98c83edddd39" containerName="maas-api" Oct 30 01:11:48.436888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.436843 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="11da6415-854c-4b21-b4da-98c83edddd39" containerName="maas-api" Oct 30 01:11:48.442604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.442582 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.465702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.465670 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-82e9bf-serving-cert\"" Oct 30 01:11:48.476914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.472700 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:11:48.518368 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.518327 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.518649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.518619 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpv8\" (UniqueName: \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.518775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.518759 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.518847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.518791 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.624660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.623298 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.624660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.623376 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpv8\" (UniqueName: \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.624660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.623509 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.624660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.623543 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.626068 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:48.625303 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-serving-cert: secret "maas-api-e2e-inf-a-82e9bf-serving-cert" not found Oct 30 01:11:48.626068 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:48.625386 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls podName:491e2c6e-3796-45d0-9746-36c11b02482b nodeName:}" failed. No retries permitted until 2026-10-30 01:11:49.125363128 +0000 UTC m=+1699.967923713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls") pod "maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" (UID: "491e2c6e-3796-45d0-9746-36c11b02482b") : secret "maas-api-e2e-inf-a-82e9bf-serving-cert" not found Oct 30 01:11:48.627750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.627700 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.653235 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.653207 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:48.659808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:48.659777 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpv8\" (UniqueName: \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:49.136196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:49.136165 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:49.138955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:49.138925 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") pod \"maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:49.358981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:49.358948 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:49.685747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:49.685721 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:11:49.686137 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:11:49.685873 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491e2c6e_3796_45d0_9746_36c11b02482b.slice/crio-dd729c3920443e4b329f291ac2709be6a0828502b98f3a2923f4f154e10dc9d8 WatchSource:0}: Error finding container dd729c3920443e4b329f291ac2709be6a0828502b98f3a2923f4f154e10dc9d8: Status 404 returned error can't find the container with id dd729c3920443e4b329f291ac2709be6a0828502b98f3a2923f4f154e10dc9d8 Oct 30 01:11:50.615342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:50.615303 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" event={"ID":"491e2c6e-3796-45d0-9746-36c11b02482b","Type":"ContainerStarted","Data":"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a"} Oct 30 01:11:50.615617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:50.615368 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" event={"ID":"491e2c6e-3796-45d0-9746-36c11b02482b","Type":"ContainerStarted","Data":"dd729c3920443e4b329f291ac2709be6a0828502b98f3a2923f4f154e10dc9d8"} Oct 30 01:11:50.615617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:50.615399 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:50.653521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:50.653461 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" podStartSLOduration=2.2880718 podStartE2EDuration="2.653444416s" totalImagesPullingTime="365.372616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:50.649236492 +0000 UTC m=+1701.491797085" watchObservedRunningTime="2026-10-30 01:11:50.653444416 +0000 UTC m=+1701.496005009" Oct 30 01:11:51.586097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:51.586070 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:11:51.925135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:51.925073 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" podUID="6dcaba65-6741-4301-9028-4e605460cec7" containerName="main" containerID="cri-o://6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486" gracePeriod=45 Oct 30 01:11:52.033940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.033135 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" podUID="6dcaba65-6741-4301-9028-4e605460cec7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": dial tcp 10.134.0.103:8000: connect: connection refused" Oct 30 01:11:52.154017 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:52.153973 5548 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcaba65_6741_4301_9028_4e605460cec7.slice/crio-6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcaba65_6741_4301_9028_4e605460cec7.slice/crio-conmon-6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:11:52.641924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.641847 5548 generic.go:356] "Generic (PLEG): container finished" podID="6dcaba65-6741-4301-9028-4e605460cec7" containerID="6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486" exitCode=0 Oct 30 01:11:52.642300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.641942 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" event={"ID":"6dcaba65-6741-4301-9028-4e605460cec7","Type":"ContainerDied","Data":"6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486"} Oct 30 01:11:52.733606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.733586 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:11:52.803087 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803056 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803135 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803172 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803227 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\" (UniqueName: \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803362 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803399 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\") pod \"6dcaba65-6741-4301-9028-4e605460cec7\" (UID: \"6dcaba65-6741-4301-9028-4e605460cec7\") " Oct 30 01:11:52.803703 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803678 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:52.803944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.803914 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:52.804290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.804266 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:52.804380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.804293 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:52.804585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.804557 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:52.806692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.806667 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:11:52.806775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.806685 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:52.806775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.806724 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h" pod "6dcaba65-6741-4301-9028-4e605460cec7" (UID: "6dcaba65-6741-4301-9028-4e605460cec7"). InnerVolumeSpecName "kube-api-access-cqc5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:52.906151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.906122 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:52.906151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.906149 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcaba65-6741-4301-9028-4e605460cec7-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:52.906151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.906159 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqc5h\" (UniqueName: \"kubernetes.io/projected/6dcaba65-6741-4301-9028-4e605460cec7-kube-api-access-cqc5h\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:52.906371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:52.906168 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcaba65-6741-4301-9028-4e605460cec7-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:53.650199 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.650173 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" Oct 30 01:11:53.650199 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.650180 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq" event={"ID":"6dcaba65-6741-4301-9028-4e605460cec7","Type":"ContainerDied","Data":"02c35072756b8e631bbb014a8572ed09bb3ddc72ddab41fddf6728c685673177"} Oct 30 01:11:53.650661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.650234 5548 scope.go:122] "RemoveContainer" containerID="6b70ad10a48c969a052f8bac12b7be3da352cb974636fbdf6412cef975bec486" Oct 30 01:11:53.712667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.711550 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:11:53.727979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.727897 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7e233f/test-model-7e233f-kserve-55d79f5d9f-kb4kq"] Oct 30 01:11:53.733278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.733249 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:11:53.733550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:53.733512 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" containerID="cri-o://8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a" gracePeriod=30 Oct 30 01:11:54.323722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.323689 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:11:54.324416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.324401 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6dcaba65-6741-4301-9028-4e605460cec7" containerName="main" Oct 30 01:11:54.324465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.324416 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcaba65-6741-4301-9028-4e605460cec7" containerName="main" Oct 30 01:11:54.324522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.324513 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcaba65-6741-4301-9028-4e605460cec7" containerName="main" Oct 30 01:11:54.328344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.328317 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.333643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.333617 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.333835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.333810 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.334012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.333991 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.334143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.334121 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77tb\" (UniqueName: \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.342691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.342670 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-56d1d9-serving-cert\"" Oct 30 01:11:54.347244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.347220 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:11:54.435154 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.435089 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77tb\" (UniqueName: \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.435302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.435245 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.435371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.435315 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.435419 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:54.435402 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-serving-cert: secret "maas-api-e2e-inf-b-56d1d9-serving-cert" not found Oct 30 01:11:54.435467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.435425 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.435521 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:54.435479 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls podName:adec1f7e-c0ee-450a-a896-c935aab0a77a nodeName:}" failed. No retries permitted until 2026-10-30 01:11:54.935460445 +0000 UTC m=+1705.778021026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls") pod "maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" (UID: "adec1f7e-c0ee-450a-a896-c935aab0a77a") : secret "maas-api-e2e-inf-b-56d1d9-serving-cert" not found Oct 30 01:11:54.436224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.436142 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.438141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.438120 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.455619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.455596 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77tb\" (UniqueName: \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.943566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.943530 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:54.946514 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:54.946485 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") pod \"maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:55.252939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:55.245561 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:55.307352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:55.307314 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcaba65-6741-4301-9028-4e605460cec7" path="/var/lib/kubelet/pods/6dcaba65-6741-4301-9028-4e605460cec7/volumes" Oct 30 01:11:55.443558 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:55.443311 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:11:55.632350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:55.632323 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:11:55.632786 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:11:55.632758 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadec1f7e_c0ee_450a_a896_c935aab0a77a.slice/crio-6930d480add8c850e6d138841bb602e54bbfdbc0e70f7731e4eb6137549d61f9 WatchSource:0}: Error finding container 6930d480add8c850e6d138841bb602e54bbfdbc0e70f7731e4eb6137549d61f9: Status 404 returned error can't find the container with id 6930d480add8c850e6d138841bb602e54bbfdbc0e70f7731e4eb6137549d61f9 Oct 30 01:11:55.675951 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:55.675812 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" event={"ID":"adec1f7e-c0ee-450a-a896-c935aab0a77a","Type":"ContainerStarted","Data":"6930d480add8c850e6d138841bb602e54bbfdbc0e70f7731e4eb6137549d61f9"} Oct 30 01:11:56.634840 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:56.634814 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:11:56.709634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:56.709581 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" event={"ID":"adec1f7e-c0ee-450a-a896-c935aab0a77a","Type":"ContainerStarted","Data":"4702f1c44795610ffde502d92e3ee37ae1f25755c011238e127d7669142af146"} Oct 30 01:11:56.711297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:56.711218 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:11:56.775122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:56.775060 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" podStartSLOduration=2.457016756 podStartE2EDuration="2.775040337s" totalImagesPullingTime="318.023581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:56.771989362 +0000 UTC m=+1707.614549954" watchObservedRunningTime="2026-10-30 01:11:56.775040337 +0000 UTC m=+1707.617600931" Oct 30 01:11:59.016530 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.016462 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:11:59.113969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.113929 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114000 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114034 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114114 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114332 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114171 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114325 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114379 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114426 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114509 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\") pod \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\" (UID: \"c5c81780-34d0-4a2c-be09-2b6e3eefe525\") " Oct 30 01:11:59.114718 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.114688 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:59.115133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115105 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:59.115253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115210 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:11:59.115324 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115039 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:59.115385 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115336 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:59.115668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115643 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.115753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115674 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.115753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115688 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.115753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115703 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.115753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.115716 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.117159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.117134 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:11:59.117387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.117342 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:11:59.118149 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.118113 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:59.119066 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.119040 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t" pod "c5c81780-34d0-4a2c-be09-2b6e3eefe525" (UID: "c5c81780-34d0-4a2c-be09-2b6e3eefe525"). InnerVolumeSpecName "kube-api-access-jqf6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:11:59.217137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.217104 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.217137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.217132 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.217137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.217142 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.217351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.217153 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqf6t\" (UniqueName: \"kubernetes.io/projected/c5c81780-34d0-4a2c-be09-2b6e3eefe525-kube-api-access-jqf6t\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:11:59.759310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.759275 5548 generic.go:356] "Generic (PLEG): container finished" podID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerID="8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a" exitCode=0 Oct 30 01:11:59.759521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.759336 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" Oct 30 01:11:59.759521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.759361 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerDied","Data":"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a"} Oct 30 01:11:59.759521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.759414 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn" event={"ID":"c5c81780-34d0-4a2c-be09-2b6e3eefe525","Type":"ContainerDied","Data":"81992b774f40bfaf38858ac7290d315d8da8614f1a729297814810014157e422"} Oct 30 01:11:59.759521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.759435 5548 scope.go:122] "RemoveContainer" containerID="8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a" Oct 30 01:11:59.770088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.770061 5548 scope.go:122] "RemoveContainer" containerID="939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0" Oct 30 01:11:59.784419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.784401 5548 scope.go:122] "RemoveContainer" containerID="8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a" Oct 30 01:11:59.784664 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:59.784645 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a\": container with ID starting with 8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a not found: ID does not exist" containerID="8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a" Oct 30 01:11:59.784711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.784674 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a"} err="failed to get container status \"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a\": rpc error: code = NotFound desc = could not find container \"8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a\": container with ID starting with 8850c86052dcfdcb2e357c7ebcb723b99583f16f7060a62fe3e8833362e7b24a not found: ID does not exist" Oct 30 01:11:59.784711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.784692 5548 scope.go:122] "RemoveContainer" containerID="939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0" Oct 30 01:11:59.785034 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:11:59.785016 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0\": container with ID starting with 939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0 not found: ID does not exist" containerID="939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0" Oct 30 01:11:59.785088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.785041 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0"} err="failed to get container status \"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0\": rpc error: code = NotFound desc = could not find container \"939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0\": container with ID starting with 939a3438bc8b4b952a987cb38867722cf22332a26e25d11004f2ae34beaa3dc0 not found: ID does not exist" Oct 30 01:11:59.817282 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.817253 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:11:59.820550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:11:59.820531 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da60de-openshift-default-64ff6c4bc-8ffkn"] Oct 30 01:12:01.267018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:01.266987 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" path="/var/lib/kubelet/pods/c5c81780-34d0-4a2c-be09-2b6e3eefe525/volumes" Oct 30 01:12:03.756495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:03.756469 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:12:07.354030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.354000 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:12:07.354638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.354622 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:07.354692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.354639 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:07.354786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.354777 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:07.354817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.354790 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:07.360078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.360059 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.379412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.379387 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:12:07.384945 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.384918 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-29145c-serving-cert\"" Oct 30 01:12:07.483074 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.483030 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.483268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.483183 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.483325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.483286 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zkgd\" (UniqueName: \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.483325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.483320 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.585265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.585224 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.585450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.585337 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.585507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.585446 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkgd\" (UniqueName: \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.585507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.585496 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.586285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.586257 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.587982 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.587951 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.588126 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.588068 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.603067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.603041 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zkgd\" (UniqueName: \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\") pod \"maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.674829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.674800 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:07.998189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:07.998154 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:12:07.998318 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:12:07.998222 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6b01a5_90fd_4827_9963_834850ee4c65.slice/crio-2e6621766adac81dba85974ea58b9b4f13aad96cdd0d0ce797272a689b99b9a4 WatchSource:0}: Error finding container 2e6621766adac81dba85974ea58b9b4f13aad96cdd0d0ce797272a689b99b9a4: Status 404 returned error can't find the container with id 2e6621766adac81dba85974ea58b9b4f13aad96cdd0d0ce797272a689b99b9a4 Oct 30 01:12:08.829774 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:08.829739 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" event={"ID":"bd6b01a5-90fd-4827-9963-834850ee4c65","Type":"ContainerStarted","Data":"35a75e3162b3302b86da033ad5f88f30539124d05e84c47849f7883a9532222f"} Oct 30 01:12:08.830239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:08.829818 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:08.830239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:08.829836 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" event={"ID":"bd6b01a5-90fd-4827-9963-834850ee4c65","Type":"ContainerStarted","Data":"2e6621766adac81dba85974ea58b9b4f13aad96cdd0d0ce797272a689b99b9a4"} Oct 30 01:12:08.867664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:08.867599 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" podStartSLOduration=1.5587304849999999 podStartE2EDuration="1.867578189s" totalImagesPullingTime="308.847704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:08.865163786 +0000 UTC m=+1719.707724378" watchObservedRunningTime="2026-10-30 01:12:08.867578189 +0000 UTC m=+1719.710138785" Oct 30 01:12:10.822799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:10.822711 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:12:10.823280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:10.822990 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" podUID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" containerName="maas-api" containerID="cri-o://fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2" gracePeriod=30 Oct 30 01:12:11.369807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.367387 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:12:11.479633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.479592 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\") pod \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " Oct 30 01:12:11.479835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.479782 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\") pod \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " Oct 30 01:12:11.479942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.479846 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\") pod \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " Oct 30 01:12:11.479942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.479923 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\" (UniqueName: \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\") pod \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\" (UID: \"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3\") " Oct 30 01:12:11.480054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.479989 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca" pod "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" (UID: "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:11.480676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.480651 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:11.482277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.482251 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls" pod "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" (UID: "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:12:11.482630 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.482602 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs" pod "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" (UID: "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:12:11.482725 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.482687 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf" pod "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" (UID: "a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3"). InnerVolumeSpecName "kube-api-access-lfgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:11.581903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.581793 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:11.581903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.581832 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:11.581903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.581848 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgpf\" (UniqueName: \"kubernetes.io/projected/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3-kube-api-access-lfgpf\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:11.856747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.856673 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" Oct 30 01:12:11.856747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.856695 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" event={"ID":"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3","Type":"ContainerDied","Data":"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2"} Oct 30 01:12:11.856747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.856623 5548 generic.go:356] "Generic (PLEG): container finished" podID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" containerID="fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2" exitCode=0 Oct 30 01:12:11.857292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.856754 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj" event={"ID":"a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3","Type":"ContainerDied","Data":"4eaa461d897f0286f4eb77f1944ac0b36a87c89a9ed7f55bfa787a680594d02d"} Oct 30 01:12:11.857292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.856775 5548 scope.go:122] "RemoveContainer" containerID="fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2" Oct 30 01:12:11.868701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.868678 5548 scope.go:122] "RemoveContainer" containerID="fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2" Oct 30 01:12:11.869077 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:11.869050 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2\": container with ID starting with fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2 not found: ID does not exist" containerID="fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2" Oct 30 01:12:11.869174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.869086 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2"} err="failed to get container status \"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2\": rpc error: code = NotFound desc = could not find container \"fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2\": container with ID starting with fa8e3f1186514773d942a2990f69837f5a843fb0fe8243103dbc5cbb90b04da2 not found: ID does not exist" Oct 30 01:12:11.903113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.903080 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:12:11.906389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:11.906370 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3204e-6ff4ff5cc6-8sbbj"] Oct 30 01:12:13.267316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:13.267277 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" path="/var/lib/kubelet/pods/a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3/volumes" Oct 30 01:12:14.842744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:14.842717 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:12:25.428193 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:25.428156 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:12:34.950065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.950033 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:12:34.950830 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.950814 5548 cpu_manager.go:414] "containerMap: removing container" podUID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" containerName="maas-api" Oct 30 01:12:34.950830 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.950831 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" containerName="maas-api" Oct 30 01:12:34.951018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.950891 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:34.951018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.950902 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c81780-34d0-4a2c-be09-2b6e3eefe525" containerName="istio-proxy" Oct 30 01:12:34.951095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.951077 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2aa4cc0-0ef7-426f-a35d-8204ff9d98f3" containerName="maas-api" Oct 30 01:12:34.956802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.956781 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:34.977162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.977133 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:12:34.977764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.977744 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52f1cb\"/\"test-model-52f1cb-kserve-self-signed-certs\"" Oct 30 01:12:34.977962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.977842 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52f1cb\"/\"default-dockercfg-2mt8d\"" Oct 30 01:12:34.995563 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.995541 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52f1cb\"/\"openshift-service-ca.crt\"" Oct 30 01:12:34.995805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:34.995785 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52f1cb\"/\"kube-root-ca.crt\"" Oct 30 01:12:35.087272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087234 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.087467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087374 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29kt\" (UniqueName: \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.087467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087422 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.087467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087447 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.087654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087588 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.087786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.087771 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.189408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.189318 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.189717 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.189693 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29kt\" (UniqueName: \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.189901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.189884 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.190025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.190012 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.190190 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.190178 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.190494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.190479 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.194881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.191050 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.220932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.195527 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.220932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.208307 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.232285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.232211 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.240542 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.233745 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29kt\" (UniqueName: \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.240542 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.234420 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\") pod \"test-model-52f1cb-kserve-64bc55d857-vtr6w\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.289906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.289567 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:35.693794 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:35.693546 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:12:35.698006 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:12:35.697962 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b242b9_5314_48b2_ac7b_9f2e28f393ca.slice/crio-9ea3216ce6ec0429f507714d8272f1eae4af385071197d8a277100608ea00623 WatchSource:0}: Error finding container 9ea3216ce6ec0429f507714d8272f1eae4af385071197d8a277100608ea00623: Status 404 returned error can't find the container with id 9ea3216ce6ec0429f507714d8272f1eae4af385071197d8a277100608ea00623 Oct 30 01:12:36.094077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:36.093964 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" event={"ID":"e1b242b9-5314-48b2-ac7b-9f2e28f393ca","Type":"ContainerStarted","Data":"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a"} Oct 30 01:12:36.094077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:36.094041 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" event={"ID":"e1b242b9-5314-48b2-ac7b-9f2e28f393ca","Type":"ContainerStarted","Data":"9ea3216ce6ec0429f507714d8272f1eae4af385071197d8a277100608ea00623"} Oct 30 01:12:36.160835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:36.160765 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" podStartSLOduration=2.160742757 podStartE2EDuration="2.160742757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:36.149278128 +0000 UTC m=+1746.991838723" watchObservedRunningTime="2026-10-30 01:12:36.160742757 +0000 UTC m=+1747.003303351" Oct 30 01:12:36.298777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:36.298745 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:38.032058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:38.032024 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:12:38.032547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:38.032237 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" containerID="cri-o://df92edefc8fc824ba21eaeb082ad85a62c81853141aa37f6458b30f5569474ca" gracePeriod=30 Oct 30 01:12:42.235196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.235159 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:12:42.239877 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.239843 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.265896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.265871 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50d078-serving-cert\"" Oct 30 01:12:42.270510 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.270481 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:12:42.331952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.331921 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2xr\" (UniqueName: \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.332156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.332073 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.332156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.332113 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.332311 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.332263 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.433641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.433604 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.433814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.433669 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.433814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.433739 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.433814 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:42.433773 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-serving-cert: secret "maas-api-e2e-inf-a-50d078-serving-cert" not found Oct 30 01:12:42.434022 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:42.433850 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls podName:cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8 nodeName:}" failed. No retries permitted until 2026-10-30 01:12:42.933830201 +0000 UTC m=+1753.776390774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls") pod "maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8") : secret "maas-api-e2e-inf-a-50d078-serving-cert" not found Oct 30 01:12:42.434089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.434031 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2xr\" (UniqueName: \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.434488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.434466 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.436319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.436296 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.454421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.454390 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2xr\" (UniqueName: \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.949112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:42.949064 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:42.949916 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:42.949492 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-serving-cert: secret "maas-api-e2e-inf-a-50d078-serving-cert" not found Oct 30 01:12:42.949916 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:42.949573 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls podName:cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8 nodeName:}" failed. No retries permitted until 2026-10-30 01:12:43.949553569 +0000 UTC m=+1754.792114165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls") pod "maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8") : secret "maas-api-e2e-inf-a-50d078-serving-cert" not found Oct 30 01:12:43.168631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.168535 5548 generic.go:356] "Generic (PLEG): container finished" podID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerID="df92edefc8fc824ba21eaeb082ad85a62c81853141aa37f6458b30f5569474ca" exitCode=0 Oct 30 01:12:43.168799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.168631 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" event={"ID":"60e6d65d-6cf9-454e-bb91-342dc4158dfb","Type":"ContainerDied","Data":"df92edefc8fc824ba21eaeb082ad85a62c81853141aa37f6458b30f5569474ca"} Oct 30 01:12:43.363663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.363630 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:12:43.459034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.458996 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459065 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459102 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459140 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459181 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459247 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459353 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459425 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.459712 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.459608 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\") pod \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\" (UID: \"60e6d65d-6cf9-454e-bb91-342dc4158dfb\") " Oct 30 01:12:43.460329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.460045 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.460765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.460466 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:43.460765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.460671 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.460765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.460691 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.461874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.461762 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.461874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.461831 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.478481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.478379 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "kube-api-access-4nfkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.478481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.478379 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:43.479009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.478976 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.484024 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.483969 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.486935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.485887 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs" pod "60e6d65d-6cf9-454e-bb91-342dc4158dfb" (UID: "60e6d65d-6cf9-454e-bb91-342dc4158dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.563155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563116 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563155 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563171 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563183 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563198 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nfkx\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-kube-api-access-4nfkx\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563215 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.563437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.563229 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e6d65d-6cf9-454e-bb91-342dc4158dfb-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.972355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:43.972272 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:44.025488 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.022464 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"maas-api-e2e-inf-a-50d078-854bb9477f-thxh4\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:44.061011 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.060971 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:44.192115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.186171 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" event={"ID":"60e6d65d-6cf9-454e-bb91-342dc4158dfb","Type":"ContainerDied","Data":"546a231c81ae7fd5fc03f1a838adce62c2220d88a44fc45a544a727b928e0335"} Oct 30 01:12:44.192115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.186236 5548 scope.go:122] "RemoveContainer" containerID="df92edefc8fc824ba21eaeb082ad85a62c81853141aa37f6458b30f5569474ca" Oct 30 01:12:44.192115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.186364 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5" Oct 30 01:12:44.246597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.246527 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:12:44.256225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.256190 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-ptrq5"] Oct 30 01:12:44.372676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:44.372647 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:12:44.373848 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:12:44.373824 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb32c2a6_8ddc_48ea_a34d_5083ea8e5cf8.slice/crio-a2249152167e8cbcce5857c9f458b01f260b203c81b5eb734ac303980f891273 WatchSource:0}: Error finding container a2249152167e8cbcce5857c9f458b01f260b203c81b5eb734ac303980f891273: Status 404 returned error can't find the container with id a2249152167e8cbcce5857c9f458b01f260b203c81b5eb734ac303980f891273 Oct 30 01:12:45.199352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.199306 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" event={"ID":"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8","Type":"ContainerStarted","Data":"5c75299519eabebbfbe4993bf3451fa2ab7dd3b808805e806f6846fabcc8b7ff"} Oct 30 01:12:45.199531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.199367 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" event={"ID":"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8","Type":"ContainerStarted","Data":"a2249152167e8cbcce5857c9f458b01f260b203c81b5eb734ac303980f891273"} Oct 30 01:12:45.199531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.199395 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:45.238015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.237968 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" podStartSLOduration=2.9212252789999997 podStartE2EDuration="3.237954179s" totalImagesPullingTime="316.7289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:45.234163757 +0000 UTC m=+1756.076724350" watchObservedRunningTime="2026-10-30 01:12:45.237954179 +0000 UTC m=+1756.080514771" Oct 30 01:12:45.266509 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.266471 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" path="/var/lib/kubelet/pods/60e6d65d-6cf9-454e-bb91-342dc4158dfb/volumes" Oct 30 01:12:45.298712 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.298682 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:45.311916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:45.311891 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:46.230001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:46.229969 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:12:51.222747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:51.222716 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:12:54.428069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:54.428032 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:12:54.428501 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:54.428111 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:12:55.437162 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:12:55.437117 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:12:56.294322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.294286 5548 generic.go:356] "Generic (PLEG): container finished" podID="d469e1b4-394b-4288-988e-7d5725f3a218" containerID="39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453" exitCode=0 Oct 30 01:12:56.294503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.294360 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerDied","Data":"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453"} Oct 30 01:12:56.294776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.294763 5548 scope.go:122] "RemoveContainer" containerID="39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453" Oct 30 01:12:56.296590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.296539 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:12:56.296655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.296630 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:12:56.296693 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:56.296661 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:12:57.307146 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:57.307102 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerStarted","Data":"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790"} Oct 30 01:12:57.724637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:57.724609 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:12:57.724828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:57.724774 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:12:57.724902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:57.724823 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:12:58.739914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:58.729227 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:12:58.739914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:58.729278 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:12:59.724658 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:59.724624 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:12:59.724955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:12:59.724929 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:13:00.724434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:00.724394 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:13:00.724965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:00.724458 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:13:01.724307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:01.724270 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:13:01.724519 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:01.724331 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:13:02.725108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:02.725075 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:13:02.725596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:02.725129 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:13:03.729827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:03.729797 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:13:03.730402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:03.730366 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:13:03.731262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:03.731241 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:13:07.344444 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.344401 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:13:07.345085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.344612 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" podUID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" containerName="maas-api" containerID="cri-o://b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224" gracePeriod=30 Oct 30 01:13:07.733242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.733206 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:13:07.867650 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.863156 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\") pod \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " Oct 30 01:13:07.867966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.867939 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") pod \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " Oct 30 01:13:07.868422 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.868401 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\") pod \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " Oct 30 01:13:07.868600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.868583 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\" (UniqueName: \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\") pod \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\" (UID: \"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19\") " Oct 30 01:13:07.886407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.885333 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca" pod "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" (UID: "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:07.920316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.920255 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88" pod "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" (UID: "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19"). InnerVolumeSpecName "kube-api-access-grh88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:07.920675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.920654 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls" pod "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" (UID: "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:07.920972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.920724 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs" pod "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" (UID: "e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:07.970597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.970564 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:07.970597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.970598 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grh88\" (UniqueName: \"kubernetes.io/projected/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-kube-api-access-grh88\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:07.970802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.970608 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:07.970802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:07.970618 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:08.407272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.407242 5548 generic.go:356] "Generic (PLEG): container finished" podID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" containerID="b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224" exitCode=0 Oct 30 01:13:08.407757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.407291 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" Oct 30 01:13:08.407757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.407331 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" event={"ID":"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19","Type":"ContainerDied","Data":"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224"} Oct 30 01:13:08.407757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.407383 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw" event={"ID":"e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19","Type":"ContainerDied","Data":"75cad0de5abe715e625cdfd2b4e853b67df099c3a90eeeeb5a33e9649a2c6f5a"} Oct 30 01:13:08.407757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.407410 5548 scope.go:122] "RemoveContainer" containerID="b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224" Oct 30 01:13:08.418732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.418715 5548 scope.go:122] "RemoveContainer" containerID="b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224" Oct 30 01:13:08.419066 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:08.419046 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224\": container with ID starting with b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224 not found: ID does not exist" containerID="b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224" Oct 30 01:13:08.419161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.419081 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224"} err="failed to get container status \"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224\": rpc error: code = NotFound desc = could not find container \"b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224\": container with ID starting with b4cc075146c7afac828985b13594195ef8d3300c46fc1e1d1aa65674e2132224 not found: ID does not exist" Oct 30 01:13:08.453152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.453123 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:13:08.463203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:08.463177 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e233f-6d658f9448-rfvhw"] Oct 30 01:13:09.322411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:09.322365 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" path="/var/lib/kubelet/pods/e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19/volumes" Oct 30 01:13:15.889916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:15.888471 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:13:15.889916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:15.888763 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" containerID="cri-o://35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d" gracePeriod=30 Oct 30 01:13:21.191641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.191602 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:13:21.301294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301261 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301316 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301341 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301447 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301533 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301604 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301679 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301678 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:21.301784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301699 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.301784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301702 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:21.301784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.301726 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\") pod \"29859192-79d3-458a-8bd5-0dea435986b2\" (UID: \"29859192-79d3-458a-8bd5-0dea435986b2\") " Oct 30 01:13:21.302255 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.302232 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:21.302746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.302724 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:21.302947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.302766 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.303192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.303177 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29859192-79d3-458a-8bd5-0dea435986b2-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.303323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.303311 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.303474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.303079 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:21.304379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.304357 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:21.304937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.304916 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "kube-api-access-2gdmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:21.305213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.305192 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:21.305270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.305221 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token" pod "29859192-79d3-458a-8bd5-0dea435986b2" (UID: "29859192-79d3-458a-8bd5-0dea435986b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:21.404937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.404899 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.405118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.404947 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.405118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.404962 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdmp\" (UniqueName: \"kubernetes.io/projected/29859192-79d3-458a-8bd5-0dea435986b2-kube-api-access-2gdmp\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.405118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.404979 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.405118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.404994 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29859192-79d3-458a-8bd5-0dea435986b2-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.405118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.405007 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29859192-79d3-458a-8bd5-0dea435986b2-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.544077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.543991 5548 generic.go:356] "Generic (PLEG): container finished" podID="29859192-79d3-458a-8bd5-0dea435986b2" containerID="35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d" exitCode=0 Oct 30 01:13:21.544077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.544044 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" Oct 30 01:13:21.544309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.544070 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" event={"ID":"29859192-79d3-458a-8bd5-0dea435986b2","Type":"ContainerDied","Data":"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d"} Oct 30 01:13:21.544309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.544132 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l" event={"ID":"29859192-79d3-458a-8bd5-0dea435986b2","Type":"ContainerDied","Data":"67bd35158b52adb8dfad858f201ab34628fcd092aba2f2461c1ed2130551563a"} Oct 30 01:13:21.544309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.544154 5548 scope.go:122] "RemoveContainer" containerID="35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d" Oct 30 01:13:21.556256 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.556232 5548 scope.go:122] "RemoveContainer" containerID="35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d" Oct 30 01:13:21.556576 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:21.556556 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d\": container with ID starting with 35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d not found: ID does not exist" containerID="35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d" Oct 30 01:13:21.556633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.556585 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d"} err="failed to get container status \"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d\": rpc error: code = NotFound desc = could not find container \"35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d\": container with ID starting with 35b8adbe840809e84be02a9f294c84ddd0475230b11328171dbdb558a3092c8d not found: ID does not exist" Oct 30 01:13:21.592688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.592655 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:13:21.602396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:21.602368 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7e233f-openshift-default-74c6dbbb65-lbk2l"] Oct 30 01:13:22.545443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.545389 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:13:22.547074 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547048 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" containerName="maas-api" Oct 30 01:13:22.547237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547221 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" containerName="maas-api" Oct 30 01:13:22.547338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547327 5548 cpu_manager.go:414] "containerMap: removing container" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" Oct 30 01:13:22.547416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547406 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" Oct 30 01:13:22.547497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547488 5548 cpu_manager.go:414] "containerMap: removing container" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" Oct 30 01:13:22.547573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.547564 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" Oct 30 01:13:22.548120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.548101 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f2c4e0-7e4b-4c5b-ac8a-7fb12639ad19" containerName="maas-api" Oct 30 01:13:22.548246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.548235 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e6d65d-6cf9-454e-bb91-342dc4158dfb" containerName="istio-proxy" Oct 30 01:13:22.548334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.548325 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="29859192-79d3-458a-8bd5-0dea435986b2" containerName="istio-proxy" Oct 30 01:13:22.555209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.555193 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.574056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.574019 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"openshift-service-ca.crt\"" Oct 30 01:13:22.577447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.577401 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:13:22.582792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.582763 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"default-dockercfg-66ftk\"" Oct 30 01:13:22.582923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.582772 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"kube-root-ca.crt\"" Oct 30 01:13:22.583000 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.582931 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"auth-test-model-64f472-kserve-self-signed-certs\"" Oct 30 01:13:22.622748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.622720 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.622953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.622778 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.623026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.622951 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.623097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.623081 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.623155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.623120 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.623213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.623173 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fh5\" (UniqueName: \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.724679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.724634 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.724911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.724703 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.724911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.724733 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fh5\" (UniqueName: \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.724911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.724829 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.725117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.724933 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.725117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.725021 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.725225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.725178 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.725768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.725406 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.725768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.725424 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.727787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.727764 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.727989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.727970 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.736164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.736137 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:13:22.740741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.740720 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.742434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.742413 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fh5\" (UniqueName: \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\") pod \"auth-test-model-64f472-kserve-7c7bffbd45-r4wwd\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.760351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.758833 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e69dee-serving-cert\"" Oct 30 01:13:22.762007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.761980 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:13:22.851291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.829138 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.851291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.829229 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.851291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.829495 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbx2\" (UniqueName: \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.851291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.829532 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.881081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.879187 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:22.930837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.930802 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbx2\" (UniqueName: \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.930837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.930849 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.931136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.930972 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.931136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.931042 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.931343 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:22.931311 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-serving-cert: secret "maas-api-e2e-inf-b-e69dee-serving-cert" not found Oct 30 01:13:22.931424 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:22.931395 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls podName:3687f115-8b30-4cbe-989e-096d7e06d3a6 nodeName:}" failed. No retries permitted until 2026-10-30 01:13:23.431373444 +0000 UTC m=+1794.273934029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls") pod "maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" (UID: "3687f115-8b30-4cbe-989e-096d7e06d3a6") : secret "maas-api-e2e-inf-b-e69dee-serving-cert" not found Oct 30 01:13:22.931998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.931964 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.934773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.934733 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:22.986912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:22.962645 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbx2\" (UniqueName: \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:23.266485 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:13:23.266455 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1599b3b_b92e_4656_9241_68148ff64461.slice/crio-064859db7774df0d7f3992da9c8ac99bca628b1edf590152c57fe0c675aff26a WatchSource:0}: Error finding container 064859db7774df0d7f3992da9c8ac99bca628b1edf590152c57fe0c675aff26a: Status 404 returned error can't find the container with id 064859db7774df0d7f3992da9c8ac99bca628b1edf590152c57fe0c675aff26a Oct 30 01:13:23.268481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.268455 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29859192-79d3-458a-8bd5-0dea435986b2" path="/var/lib/kubelet/pods/29859192-79d3-458a-8bd5-0dea435986b2/volumes" Oct 30 01:13:23.268933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.268915 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:13:23.440653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.440615 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:23.443388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.443369 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") pod \"maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:23.597891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.597787 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" event={"ID":"c1599b3b-b92e-4656-9241-68148ff64461","Type":"ContainerStarted","Data":"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2"} Oct 30 01:13:23.597891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.597835 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" event={"ID":"c1599b3b-b92e-4656-9241-68148ff64461","Type":"ContainerStarted","Data":"064859db7774df0d7f3992da9c8ac99bca628b1edf590152c57fe0c675aff26a"} Oct 30 01:13:23.633750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.633690 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" podStartSLOduration=1.633672818 podStartE2EDuration="1.633672818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:23.63182159 +0000 UTC m=+1794.474382200" watchObservedRunningTime="2026-10-30 01:13:23.633672818 +0000 UTC m=+1794.476233411" Oct 30 01:13:23.736923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.736328 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:23.882053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:23.882023 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:24.100981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:24.100922 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:13:24.102483 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:13:24.102453 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3687f115_8b30_4cbe_989e_096d7e06d3a6.slice/crio-ea47d203435ef00659ded3b905c83373eac078922993d23d6256d3ff14e34993 WatchSource:0}: Error finding container ea47d203435ef00659ded3b905c83373eac078922993d23d6256d3ff14e34993: Status 404 returned error can't find the container with id ea47d203435ef00659ded3b905c83373eac078922993d23d6256d3ff14e34993 Oct 30 01:13:24.606807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:24.606770 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" event={"ID":"3687f115-8b30-4cbe-989e-096d7e06d3a6","Type":"ContainerStarted","Data":"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1"} Oct 30 01:13:24.607236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:24.606827 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" event={"ID":"3687f115-8b30-4cbe-989e-096d7e06d3a6","Type":"ContainerStarted","Data":"ea47d203435ef00659ded3b905c83373eac078922993d23d6256d3ff14e34993"} Oct 30 01:13:24.607236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:24.606893 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:24.643279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:24.643164 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" podStartSLOduration=2.256411421 podStartE2EDuration="2.643149193s" totalImagesPullingTime="386.737772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:24.641664626 +0000 UTC m=+1795.484225218" watchObservedRunningTime="2026-10-30 01:13:24.643149193 +0000 UTC m=+1795.485709840" Oct 30 01:13:25.430148 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:25.430109 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:13:29.601122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:29.601092 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:13:29.603444 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:29.603421 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:13:29.956885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:29.953240 5548 scope.go:122] "RemoveContainer" containerID="781406937bc9ba52c0a95e7f5c280b81b54f086a3429b226c07dae0d9b172f9f" Oct 30 01:13:30.621582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:30.621556 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:13:32.881880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:32.881837 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:32.895581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:32.895557 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:33.869962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:33.869649 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:13:38.013936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:38.013892 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:13:38.014428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:38.014204 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" containerID="cri-o://e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b" gracePeriod=30 Oct 30 01:13:38.443612 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:38.443563 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 01:13:38.443888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:38.443841 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" containerID="cri-o://ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162" gracePeriod=30 Oct 30 01:13:43.289087 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.289026 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:13:43.341995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.341963 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.342175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342024 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.342175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342077 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.342175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342126 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342224 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342262 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342326 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342349 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342439 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342483 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.342545 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.343153 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.343203 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\") pod \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\" (UID: \"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8\") " Oct 30 01:13:43.343875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.343294 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.345487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.345464 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.348946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.345491 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.348946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.345505 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.348946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.345517 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.348946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.345530 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.348946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.346018 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.349960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.349938 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:43.362386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.362351 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:43.362498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.362408 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw" pod "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" (UID: "7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8"). InnerVolumeSpecName "kube-api-access-bkjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:43.446743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.446706 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjzw\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-kube-api-access-bkjzw\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.446743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.446739 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.446743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.446748 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.446743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.446756 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.726494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.726470 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 01:13:43.751261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751228 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751281 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751388 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751452 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751485 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751526 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751571 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751753 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751791 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751805 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\") pod \"d9c42d62-0275-417c-932e-de81098d49df\" (UID: \"d9c42d62-0275-417c-932e-de81098d49df\") " Oct 30 01:13:43.751997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.751813 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.752549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752158 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:43.752549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752281 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.752651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752555 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.752651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752592 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9c42d62-0275-417c-932e-de81098d49df-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.752651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752608 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.752651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.752622 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.753094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.753070 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.754698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.754568 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:43.754698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.754666 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "kube-api-access-55nvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:43.754932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.754717 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.755074 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.755053 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo" pod "d9c42d62-0275-417c-932e-de81098d49df" (UID: "d9c42d62-0275-417c-932e-de81098d49df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:43.854601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.854517 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.854601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.854548 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.854601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.854557 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9c42d62-0275-417c-932e-de81098d49df-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.854601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.854565 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9c42d62-0275-417c-932e-de81098d49df-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.854601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.854573 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55nvl\" (UniqueName: \"kubernetes.io/projected/d9c42d62-0275-417c-932e-de81098d49df-kube-api-access-55nvl\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.855941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.855911 5548 generic.go:356] "Generic (PLEG): container finished" podID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerID="e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b" exitCode=0 Oct 30 01:13:43.856077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.855963 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" Oct 30 01:13:43.856077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.855993 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" event={"ID":"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8","Type":"ContainerDied","Data":"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b"} Oct 30 01:13:43.856077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.856057 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk" event={"ID":"7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8","Type":"ContainerDied","Data":"d0bf0c49007985a6bbd4831aaf0c898f1ba8ae7c1d69a82971a203d94eecc231"} Oct 30 01:13:43.856227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.856087 5548 scope.go:122] "RemoveContainer" containerID="e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b" Oct 30 01:13:43.857726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.857701 5548 generic.go:356] "Generic (PLEG): container finished" podID="d9c42d62-0275-417c-932e-de81098d49df" containerID="ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162" exitCode=0 Oct 30 01:13:43.858004 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.857984 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" event={"ID":"d9c42d62-0275-417c-932e-de81098d49df","Type":"ContainerDied","Data":"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162"} Oct 30 01:13:43.858073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.858031 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" event={"ID":"d9c42d62-0275-417c-932e-de81098d49df","Type":"ContainerDied","Data":"0a7106f549eed91ee0437514e760c280546bc1efd2483c09c8b60bd5420aa2f6"} Oct 30 01:13:43.858120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.858105 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46" Oct 30 01:13:43.868584 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.868564 5548 scope.go:122] "RemoveContainer" containerID="e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b" Oct 30 01:13:43.868978 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:43.868945 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b\": container with ID starting with e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b not found: ID does not exist" containerID="e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b" Oct 30 01:13:43.869094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.868989 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b"} err="failed to get container status \"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b\": rpc error: code = NotFound desc = could not find container \"e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b\": container with ID starting with e7dacd26574d422d15abe14a092d9be42a301e0f65ab9660b570e642ea9b965b not found: ID does not exist" Oct 30 01:13:43.869094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.869010 5548 scope.go:122] "RemoveContainer" containerID="ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162" Oct 30 01:13:43.879719 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.879697 5548 scope.go:122] "RemoveContainer" containerID="ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162" Oct 30 01:13:43.880062 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:43.880041 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162\": container with ID starting with ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162 not found: ID does not exist" containerID="ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162" Oct 30 01:13:43.880154 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.880068 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162"} err="failed to get container status \"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162\": rpc error: code = NotFound desc = could not find container \"ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162\": container with ID starting with ae089a2e1784473d28f3183d77fc3cf4f0fc787db59c743529bcc568410bd162 not found: ID does not exist" Oct 30 01:13:43.915084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.915054 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:13:43.918386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.918363 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-vzwxk"] Oct 30 01:13:43.946630 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.946599 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 01:13:43.950027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:43.949998 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-6mf46"] Oct 30 01:13:45.266432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:45.266399 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" path="/var/lib/kubelet/pods/7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8/volumes" Oct 30 01:13:45.267175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:13:45.267152 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c42d62-0275-417c-932e-de81098d49df" path="/var/lib/kubelet/pods/d9c42d62-0275-417c-932e-de81098d49df/volumes" Oct 30 01:13:55.430527 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:13:55.430489 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:14:02.861942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.861904 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:14:02.862543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862528 5548 cpu_manager.go:414] "containerMap: removing container" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" Oct 30 01:14:02.862592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862543 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" Oct 30 01:14:02.862592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862582 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" Oct 30 01:14:02.862592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862587 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" Oct 30 01:14:02.862685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862678 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c42d62-0275-417c-932e-de81098d49df" containerName="istio-proxy" Oct 30 01:14:02.862728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.862699 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6672ba-a11c-4a3e-b9a2-3b9562e6cce8" containerName="istio-proxy" Oct 30 01:14:02.873911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.871494 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.906313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.906279 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:14:02.926916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.926883 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927071 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927048 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927080 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927110 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927132 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927280 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927367 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927315 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927406 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927369 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.927438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.927410 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2g4\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:02.991114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:02.991081 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-687471-openshift-default-dockercfg-nwb7x\"" Oct 30 01:14:03.029061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029027 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029165 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029205 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029317 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029252 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029317 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029278 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029333 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029360 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029396 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029457 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2g4\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029636 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029682 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029788 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029728 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.029926 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029904 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.030001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.029952 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.031915 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.031889 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.032135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.032115 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.052013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.051985 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2g4\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.052409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.052391 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\") pod \"e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.187698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.187670 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:03.507077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.507050 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:14:03.508495 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:14:03.508472 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c2373c_0ab6_4b45_b34d_39165dc21d94.slice/crio-7403f1006250f1840e0c64ef3b08b0f8781444ffe93d62828cb8ab2f915724ac WatchSource:0}: Error finding container 7403f1006250f1840e0c64ef3b08b0f8781444ffe93d62828cb8ab2f915724ac: Status 404 returned error can't find the container with id 7403f1006250f1840e0c64ef3b08b0f8781444ffe93d62828cb8ab2f915724ac Oct 30 01:14:03.511349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.511315 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:03.511445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.511383 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:03.511445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:03.511411 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:04.006472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.006438 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" event={"ID":"b0c2373c-0ab6-4b45-b34d-39165dc21d94","Type":"ContainerStarted","Data":"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4"} Oct 30 01:14:04.006963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.006489 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" event={"ID":"b0c2373c-0ab6-4b45-b34d-39165dc21d94","Type":"ContainerStarted","Data":"7403f1006250f1840e0c64ef3b08b0f8781444ffe93d62828cb8ab2f915724ac"} Oct 30 01:14:04.066921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.066872 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podStartSLOduration=2.0668419670000002 podStartE2EDuration="2.066841967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:04.05311346 +0000 UTC m=+1834.895674054" watchObservedRunningTime="2026-10-30 01:14:04.066841967 +0000 UTC m=+1834.909402559" Oct 30 01:14:04.188072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.188042 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:04.189702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.189678 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:04.189798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:04.189726 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:05.188358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:05.188321 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:05.188991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:05.188944 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:06.188884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:06.188314 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:06.188884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:06.188391 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:07.189884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:07.189240 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:07.189884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:07.189318 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:08.195884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:08.195292 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:08.195884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:08.195371 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:09.191881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:09.191271 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:09.191881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:09.191355 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:10.188590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:10.188552 5548 patch_prober.go:29] interesting pod/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:14:10.189116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:10.188617 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 01:14:11.193166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:11.193132 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:11.193644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:11.193608 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:11.194298 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:11.194280 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:14:20.130709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.130673 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:14:20.139475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.138249 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.156203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.156175 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0e5711-serving-cert\"" Oct 30 01:14:20.175456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.175409 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:14:20.359743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.359516 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.359964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.359746 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw99m\" (UniqueName: \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.360075 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.360045 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.360136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.360120 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.461296 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.461264 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.461494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.461318 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.461494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.461366 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.461494 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:20.461447 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-serving-cert: secret "maas-api-e2e-inf-b-0e5711-serving-cert" not found Oct 30 01:14:20.461661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.461499 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw99m\" (UniqueName: \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.461661 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:20.461522 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls podName:c4e3ac13-083a-4dfa-905e-56c5ae912036 nodeName:}" failed. No retries permitted until 2026-10-30 01:14:20.96150134 +0000 UTC m=+1851.804061930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls") pod "maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036") : secret "maas-api-e2e-inf-b-0e5711-serving-cert" not found Oct 30 01:14:20.462265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.462190 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.464113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.464090 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.479239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.479217 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw99m\" (UniqueName: \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.967563 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:20.967519 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:20.967764 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:20.967670 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-serving-cert: secret "maas-api-e2e-inf-b-0e5711-serving-cert" not found Oct 30 01:14:20.967764 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:20.967736 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls podName:c4e3ac13-083a-4dfa-905e-56c5ae912036 nodeName:}" failed. No retries permitted until 2026-10-30 01:14:21.96772262 +0000 UTC m=+1852.810283207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls") pod "maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036") : secret "maas-api-e2e-inf-b-0e5711-serving-cert" not found Oct 30 01:14:21.984344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:21.984304 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:21.987086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:21.987064 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:22.257402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:22.257329 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:22.566663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:22.565746 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:14:22.567723 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:14:22.567683 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e3ac13_083a_4dfa_905e_56c5ae912036.slice/crio-96efc58cf68251abdd93b8c93786831eefa8aa7d91e5dfc137c06f9a54730daa WatchSource:0}: Error finding container 96efc58cf68251abdd93b8c93786831eefa8aa7d91e5dfc137c06f9a54730daa: Status 404 returned error can't find the container with id 96efc58cf68251abdd93b8c93786831eefa8aa7d91e5dfc137c06f9a54730daa Oct 30 01:14:23.218541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:23.218509 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" event={"ID":"c4e3ac13-083a-4dfa-905e-56c5ae912036","Type":"ContainerStarted","Data":"fced27f451aed375a4306b4438efc004f430ce7bca5dcb544e6beaaac5a230b6"} Oct 30 01:14:23.219031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:23.218559 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" event={"ID":"c4e3ac13-083a-4dfa-905e-56c5ae912036","Type":"ContainerStarted","Data":"96efc58cf68251abdd93b8c93786831eefa8aa7d91e5dfc137c06f9a54730daa"} Oct 30 01:14:23.219031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:23.218597 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:23.255464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:23.255215 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" podStartSLOduration=2.812265221 podStartE2EDuration="3.255194808s" totalImagesPullingTime="442.929587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:23.25319779 +0000 UTC m=+1854.095758383" watchObservedRunningTime="2026-10-30 01:14:23.255194808 +0000 UTC m=+1854.097755402" Oct 30 01:14:25.428462 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:25.428430 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:14:29.267431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:29.267401 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:14:44.216048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.216012 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:14:44.222663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.222640 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.238473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.238436 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-eacea3-openshift-default-dockercfg-bqqn7\"" Oct 30 01:14:44.258916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.252463 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:14:44.333391 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333345 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333465 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333505 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333627 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333662 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzql\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333754 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.333908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.333893 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.334027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.334011 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.334083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.334062 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.435880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.435835 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.435911 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzql\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.435949 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436035 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436282 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436142 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436282 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436193 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436338 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436412 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436439 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436448 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.436601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436470 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.437117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436722 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.437117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.436808 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.437352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.437324 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.438728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.438703 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.439004 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.438987 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.453498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.453470 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.453599 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.453578 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzql\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.536995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.536916 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:44.729747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.729714 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:14:44.738758 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.738728 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.758544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.758506 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:14:44.837344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.837253 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-64f472-openshift-default-dockercfg-vx8t4\"" Oct 30 01:14:44.841991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.841959 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842089 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842141 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hz6\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842177 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842260 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842294 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842451 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842556 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.842638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.842599 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.865671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.865630 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:14:44.870445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.870405 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:44.870564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.870479 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:44.870564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.870510 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:44.943587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943560 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.943715 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943604 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.943771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943730 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.943849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943805 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.943849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943841 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944008 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.943994 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944070 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944110 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hz6\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944144 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944154 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944183 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944483 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944557 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.944668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.944648 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.947432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.947407 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.947649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.947634 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.971234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.971201 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hz6\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:44.998886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:44.998177 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\") pod \"e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:45.070379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.070078 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:45.423189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.423154 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" event={"ID":"e4df4f73-9097-424d-a793-ccd0c1fb6b48","Type":"ContainerStarted","Data":"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d"} Oct 30 01:14:45.423618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.423224 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" event={"ID":"e4df4f73-9097-424d-a793-ccd0c1fb6b48","Type":"ContainerStarted","Data":"d05e38a669f8eceef4df649e2ba897bbd2415b38d880fcf8e9ddc0945e0c67b6"} Oct 30 01:14:45.428759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.428727 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:14:45.429510 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:14:45.429475 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a34cd0e_2853_426b_a535_ae50e90bf222.slice/crio-9deca290fd26d7e841219a889c8315e194a7081bef96cf2cfcb74da5e1d9f31b WatchSource:0}: Error finding container 9deca290fd26d7e841219a889c8315e194a7081bef96cf2cfcb74da5e1d9f31b: Status 404 returned error can't find the container with id 9deca290fd26d7e841219a889c8315e194a7081bef96cf2cfcb74da5e1d9f31b Oct 30 01:14:45.432589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.432551 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:45.432678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.432621 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:45.432678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.432649 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:14:45.463370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.463313 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podStartSLOduration=1.463296767 podStartE2EDuration="1.463296767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:45.458989656 +0000 UTC m=+1876.301550253" watchObservedRunningTime="2026-10-30 01:14:45.463296767 +0000 UTC m=+1876.305857360" Oct 30 01:14:45.538035 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.538008 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:45.539749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.539726 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:45.539886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:45.539769 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:46.431767 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:46.431728 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" event={"ID":"3a34cd0e-2853-426b-a535-ae50e90bf222","Type":"ContainerStarted","Data":"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4"} Oct 30 01:14:46.432292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:46.431796 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" event={"ID":"3a34cd0e-2853-426b-a535-ae50e90bf222","Type":"ContainerStarted","Data":"9deca290fd26d7e841219a889c8315e194a7081bef96cf2cfcb74da5e1d9f31b"} Oct 30 01:14:46.493359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:46.493300 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podStartSLOduration=2.493280163 podStartE2EDuration="2.493280163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:46.491238755 +0000 UTC m=+1877.333799361" watchObservedRunningTime="2026-10-30 01:14:46.493280163 +0000 UTC m=+1877.335840757" Oct 30 01:14:46.537531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:46.537494 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:46.537759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:46.537738 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:47.074335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:47.072211 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:47.078883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:47.075765 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:47.078883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:47.075822 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:47.544887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:47.537331 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:47.544887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:47.537397 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:48.071885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:48.071266 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:48.071885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:48.071330 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:48.544528 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:48.544492 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:48.544812 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:48.544786 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:49.072889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:49.072020 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:49.072889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:49.072092 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:49.537985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:49.537944 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:49.538157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:49.537999 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:50.076959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:50.071272 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:50.076959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:50.071339 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:50.537526 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:50.537488 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:50.537708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:50.537556 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:51.071886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:51.071238 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:51.071886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:51.071298 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:51.537371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:51.537330 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:14:51.538038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:51.537407 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 01:14:52.072886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:52.072243 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:14:52.072886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:52.072315 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 01:14:52.542603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:52.542569 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:52.543144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:52.543114 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:52.543821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:52.543801 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:14:53.076231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:53.076204 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:53.076704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:53.076685 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:53.077476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:14:53.077457 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:14:55.430991 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:14:55.430952 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:15:00.336556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.336516 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:00.341805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.341781 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.357191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.357145 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:00.361407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.361384 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:15:00.382576 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.382546 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:00.382748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.382590 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:00.382748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.382662 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.467468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.467427 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzts\" (UniqueName: \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.467644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.467623 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.467709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.467694 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4dv\" (UniqueName: \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.467870 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.467841 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.568966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.568936 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4dv\" (UniqueName: \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.569177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.569067 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.569177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.569142 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzts\" (UniqueName: \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.569320 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.569302 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.570178 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.570156 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.570306 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.570180 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.592779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.592712 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4dv\" (UniqueName: \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\") pod \"maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.592911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.592796 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzts\" (UniqueName: \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\") pod \"maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.667241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.667210 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:00.705595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.705564 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:00.986090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:00.986063 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:00.986826 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:15:00.986802 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dbbe9d_f2c7_4a8a_8096_03c3c43540f6.slice/crio-40db99fbf6e1658ec7e8e239b9284fbedba7671557451c41a7e85e5e0ba82103 WatchSource:0}: Error finding container 40db99fbf6e1658ec7e8e239b9284fbedba7671557451c41a7e85e5e0ba82103: Status 404 returned error can't find the container with id 40db99fbf6e1658ec7e8e239b9284fbedba7671557451c41a7e85e5e0ba82103 Oct 30 01:15:01.059511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.059486 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:01.060488 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:15:01.060414 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82daacfd_5e9b_4f41_847d_d7e7c3fa7db6.slice/crio-54e018a10985e0abecd593cf1bd853a22864c48547e375b5f8d693c8c41dc5d8 WatchSource:0}: Error finding container 54e018a10985e0abecd593cf1bd853a22864c48547e375b5f8d693c8c41dc5d8: Status 404 returned error can't find the container with id 54e018a10985e0abecd593cf1bd853a22864c48547e375b5f8d693c8c41dc5d8 Oct 30 01:15:01.626123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.626091 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerStarted","Data":"354caae7d1913ba1571022accbe48afc215bf07952be6c2197c0365c8eb97b2f"} Oct 30 01:15:01.626535 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.626145 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerStarted","Data":"40db99fbf6e1658ec7e8e239b9284fbedba7671557451c41a7e85e5e0ba82103"} Oct 30 01:15:01.627976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.627949 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerStarted","Data":"777e4769d17d0d588928b4cc23fb4eefafc99b980698079fbd5a4932c4272f0e"} Oct 30 01:15:01.628090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.627997 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerStarted","Data":"54e018a10985e0abecd593cf1bd853a22864c48547e375b5f8d693c8c41dc5d8"} Oct 30 01:15:01.664753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.664704 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" podStartSLOduration=1.6646838000000002 podStartE2EDuration="1.6646838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.661083682 +0000 UTC m=+1892.503644276" watchObservedRunningTime="2026-10-30 01:15:01.6646838 +0000 UTC m=+1892.507244399" Oct 30 01:15:01.699124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:01.699066 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" podStartSLOduration=1.69905238 podStartE2EDuration="1.69905238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.696840089 +0000 UTC m=+1892.539400685" watchObservedRunningTime="2026-10-30 01:15:01.69905238 +0000 UTC m=+1892.541612965" Oct 30 01:15:02.637433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.637395 5548 generic.go:356] "Generic (PLEG): container finished" podID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerID="354caae7d1913ba1571022accbe48afc215bf07952be6c2197c0365c8eb97b2f" exitCode=60 Oct 30 01:15:02.638372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.637434 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerDied","Data":"354caae7d1913ba1571022accbe48afc215bf07952be6c2197c0365c8eb97b2f"} Oct 30 01:15:02.638372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.637786 5548 scope.go:122] "RemoveContainer" containerID="354caae7d1913ba1571022accbe48afc215bf07952be6c2197c0365c8eb97b2f" Oct 30 01:15:02.639527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.639499 5548 generic.go:356] "Generic (PLEG): container finished" podID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerID="777e4769d17d0d588928b4cc23fb4eefafc99b980698079fbd5a4932c4272f0e" exitCode=60 Oct 30 01:15:02.639771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.639745 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerDied","Data":"777e4769d17d0d588928b4cc23fb4eefafc99b980698079fbd5a4932c4272f0e"} Oct 30 01:15:02.639924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:02.639792 5548 scope.go:122] "RemoveContainer" containerID="777e4769d17d0d588928b4cc23fb4eefafc99b980698079fbd5a4932c4272f0e" Oct 30 01:15:03.648122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:03.648085 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerStarted","Data":"80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666"} Oct 30 01:15:03.650278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:03.650250 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerStarted","Data":"dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d"} Oct 30 01:15:04.660616 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.660580 5548 generic.go:356] "Generic (PLEG): container finished" podID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerID="80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666" exitCode=60 Oct 30 01:15:04.661132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.660618 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerDied","Data":"80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666"} Oct 30 01:15:04.661132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.660682 5548 scope.go:122] "RemoveContainer" containerID="354caae7d1913ba1571022accbe48afc215bf07952be6c2197c0365c8eb97b2f" Oct 30 01:15:04.661132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.661044 5548 scope.go:122] "RemoveContainer" containerID="80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666" Oct 30 01:15:04.661445 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:04.661421 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p_odh-ai-gateway-infra(01dbbe9d-f2c7-4a8a-8096-03c3c43540f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" Oct 30 01:15:04.663043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.663005 5548 generic.go:356] "Generic (PLEG): container finished" podID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerID="dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d" exitCode=60 Oct 30 01:15:04.663149 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.663079 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerDied","Data":"dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d"} Oct 30 01:15:04.663411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.663392 5548 scope.go:122] "RemoveContainer" containerID="dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d" Oct 30 01:15:04.663725 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:04.663706 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2_odh-ai-gateway-infra(82daacfd-5e9b-4f41-847d-d7e7c3fa7db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" Oct 30 01:15:04.674497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:04.674478 5548 scope.go:122] "RemoveContainer" containerID="777e4769d17d0d588928b4cc23fb4eefafc99b980698079fbd5a4932c4272f0e" Oct 30 01:15:05.672600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:05.672567 5548 scope.go:122] "RemoveContainer" containerID="80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666" Oct 30 01:15:05.673147 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:05.672989 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p_odh-ai-gateway-infra(01dbbe9d-f2c7-4a8a-8096-03c3c43540f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" Oct 30 01:15:05.679751 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:05.679726 5548 scope.go:122] "RemoveContainer" containerID="dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d" Oct 30 01:15:05.680209 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:05.680186 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2_odh-ai-gateway-infra(82daacfd-5e9b-4f41-847d-d7e7c3fa7db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" Oct 30 01:15:15.257242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:15.257213 5548 scope.go:122] "RemoveContainer" containerID="dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d" Oct 30 01:15:15.761358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:15.761320 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerStarted","Data":"6ac2059c3b96a4ad800bad688fa51a8603361953a1df88c2a5bdf14935404626"} Oct 30 01:15:16.774228 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:16.774196 5548 generic.go:356] "Generic (PLEG): container finished" podID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerID="6ac2059c3b96a4ad800bad688fa51a8603361953a1df88c2a5bdf14935404626" exitCode=60 Oct 30 01:15:16.774707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:16.774268 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerDied","Data":"6ac2059c3b96a4ad800bad688fa51a8603361953a1df88c2a5bdf14935404626"} Oct 30 01:15:16.774707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:16.774327 5548 scope.go:122] "RemoveContainer" containerID="dc1514ded6db6a9ee4613a7d38a81e6474b2c062fd43bfca2f457e75d1acff8d" Oct 30 01:15:16.774841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:16.774773 5548 scope.go:122] "RemoveContainer" containerID="6ac2059c3b96a4ad800bad688fa51a8603361953a1df88c2a5bdf14935404626" Oct 30 01:15:16.775281 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:16.775259 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2_odh-ai-gateway-infra(82daacfd-5e9b-4f41-847d-d7e7c3fa7db6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" Oct 30 01:15:16.806277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:16.806244 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:17.956230 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:17.956204 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:18.030676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.030636 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\" (UniqueName: \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\") pod \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " Oct 30 01:15:18.030887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.030693 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\") pod \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\" (UID: \"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6\") " Oct 30 01:15:18.031172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.031140 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca" pod "82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" (UID: "82daacfd-5e9b-4f41-847d-d7e7c3fa7db6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:18.031528 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.031473 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.034125 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.034096 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts" pod "82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" (UID: "82daacfd-5e9b-4f41-847d-d7e7c3fa7db6"). InnerVolumeSpecName "kube-api-access-mpzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:18.132629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.132593 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzts\" (UniqueName: \"kubernetes.io/projected/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6-kube-api-access-mpzts\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:15:18.257761 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.257729 5548 scope.go:122] "RemoveContainer" containerID="80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666" Oct 30 01:15:18.801195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.801161 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerStarted","Data":"e0c543d3b4f3a4c5ddb232ae16f0539836c78e03d30c02d9997c895950c177ea"} Oct 30 01:15:18.802647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.802620 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" Oct 30 01:15:18.802783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.802624 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2" event={"ID":"82daacfd-5e9b-4f41-847d-d7e7c3fa7db6","Type":"ContainerDied","Data":"54e018a10985e0abecd593cf1bd853a22864c48547e375b5f8d693c8c41dc5d8"} Oct 30 01:15:18.802783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.802716 5548 scope.go:122] "RemoveContainer" containerID="6ac2059c3b96a4ad800bad688fa51a8603361953a1df88c2a5bdf14935404626" Oct 30 01:15:18.869665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.869583 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:18.873125 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:18.873101 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0e5711-29888715-z7tn2"] Oct 30 01:15:19.265809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.265768 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" path="/var/lib/kubelet/pods/82daacfd-5e9b-4f41-847d-d7e7c3fa7db6/volumes" Oct 30 01:15:19.812120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.812086 5548 generic.go:356] "Generic (PLEG): container finished" podID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerID="e0c543d3b4f3a4c5ddb232ae16f0539836c78e03d30c02d9997c895950c177ea" exitCode=60 Oct 30 01:15:19.812312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.812153 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerDied","Data":"e0c543d3b4f3a4c5ddb232ae16f0539836c78e03d30c02d9997c895950c177ea"} Oct 30 01:15:19.812312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.812209 5548 scope.go:122] "RemoveContainer" containerID="80aace7e3182482f605fa8ede70e50d7ca983be66e3d385944d193521de87666" Oct 30 01:15:19.812660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.812631 5548 scope.go:122] "RemoveContainer" containerID="e0c543d3b4f3a4c5ddb232ae16f0539836c78e03d30c02d9997c895950c177ea" Oct 30 01:15:19.813088 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:19.813064 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p_odh-ai-gateway-infra(01dbbe9d-f2c7-4a8a-8096-03c3c43540f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" Oct 30 01:15:19.849015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:19.848987 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:20.990053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:20.990027 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:21.076623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.076539 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\") pod \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " Oct 30 01:15:21.076623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.076607 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\" (UniqueName: \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\") pod \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\" (UID: \"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6\") " Oct 30 01:15:21.076981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.076952 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca" pod "01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" (UID: "01dbbe9d-f2c7-4a8a-8096-03c3c43540f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:21.077671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.077642 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.078990 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.078961 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv" pod "01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" (UID: "01dbbe9d-f2c7-4a8a-8096-03c3c43540f6"). InnerVolumeSpecName "kube-api-access-rj4dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:21.179637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.179605 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj4dv\" (UniqueName: \"kubernetes.io/projected/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6-kube-api-access-rj4dv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.832097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.832063 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" Oct 30 01:15:21.832333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.832103 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p" event={"ID":"01dbbe9d-f2c7-4a8a-8096-03c3c43540f6","Type":"ContainerDied","Data":"40db99fbf6e1658ec7e8e239b9284fbedba7671557451c41a7e85e5e0ba82103"} Oct 30 01:15:21.832333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.832214 5548 scope.go:122] "RemoveContainer" containerID="e0c543d3b4f3a4c5ddb232ae16f0539836c78e03d30c02d9997c895950c177ea" Oct 30 01:15:21.879398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.879363 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:21.890674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:21.890645 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e92dab-29888715-rj89p"] Oct 30 01:15:23.269201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:15:23.269158 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" path="/var/lib/kubelet/pods/01dbbe9d-f2c7-4a8a-8096-03c3c43540f6/volumes" Oct 30 01:15:25.429321 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:25.429278 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:15:55.429751 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:15:55.429712 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:16:13.250885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:13.232874 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:16:25.428122 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:16:25.428078 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:16:28.276434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.276155 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" podUID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" containerName="main" containerID="cri-o://e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a" gracePeriod=45 Oct 30 01:16:28.711826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.711459 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:16:28.790209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790160 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.790419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790217 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\" (UniqueName: \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.790419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790268 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.790560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790443 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.790659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790643 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.790739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.790725 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\") pod \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\" (UID: \"e1b242b9-5314-48b2-ac7b-9f2e28f393ca\") " Oct 30 01:16:28.791243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.791214 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:28.791243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.791227 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:28.791568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.791399 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:28.791568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.791420 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:28.791568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.791459 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:28.794533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.794505 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:28.798612 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.798589 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:28.805852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.805827 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt" pod "e1b242b9-5314-48b2-ac7b-9f2e28f393ca" (UID: "e1b242b9-5314-48b2-ac7b-9f2e28f393ca"). InnerVolumeSpecName "kube-api-access-f29kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:28.892660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.892628 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:28.892660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.892658 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:28.892852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.892670 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29kt\" (UniqueName: \"kubernetes.io/projected/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-kube-api-access-f29kt\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:28.892852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:28.892682 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b242b9-5314-48b2-ac7b-9f2e28f393ca-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:29.374024 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.373987 5548 generic.go:356] "Generic (PLEG): container finished" podID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" containerID="e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a" exitCode=0 Oct 30 01:16:29.374448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.374058 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" Oct 30 01:16:29.374448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.374070 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" event={"ID":"e1b242b9-5314-48b2-ac7b-9f2e28f393ca","Type":"ContainerDied","Data":"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a"} Oct 30 01:16:29.374448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.374123 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w" event={"ID":"e1b242b9-5314-48b2-ac7b-9f2e28f393ca","Type":"ContainerDied","Data":"9ea3216ce6ec0429f507714d8272f1eae4af385071197d8a277100608ea00623"} Oct 30 01:16:29.374448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.374141 5548 scope.go:122] "RemoveContainer" containerID="e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a" Oct 30 01:16:29.385456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.385434 5548 scope.go:122] "RemoveContainer" containerID="e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a" Oct 30 01:16:29.385727 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:16:29.385701 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a\": container with ID starting with e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a not found: ID does not exist" containerID="e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a" Oct 30 01:16:29.385799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.385735 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a"} err="failed to get container status \"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a\": rpc error: code = NotFound desc = could not find container \"e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a\": container with ID starting with e9878d539c5f6c4c370c905ccf1d43db50622fab68e48a1c5dc9d4f31c89f86a not found: ID does not exist" Oct 30 01:16:29.419231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.419197 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:16:29.428805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:29.428779 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-52f1cb/test-model-52f1cb-kserve-64bc55d857-vtr6w"] Oct 30 01:16:31.265919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:31.265888 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" path="/var/lib/kubelet/pods/e1b242b9-5314-48b2-ac7b-9f2e28f393ca/volumes" Oct 30 01:16:33.431677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.431638 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8753-e71e23"] Oct 30 01:16:33.432359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432343 5548 cpu_manager.go:414] "containerMap: removing container" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432360 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432371 5548 cpu_manager.go:414] "containerMap: removing container" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432376 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432405 5548 cpu_manager.go:414] "containerMap: removing container" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432410 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432417 5548 cpu_manager.go:414] "containerMap: removing container" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432422 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432436 5548 cpu_manager.go:414] "containerMap: removing container" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432440 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432463 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" containerName="main" Oct 30 01:16:33.432538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432467 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" containerName="main" Oct 30 01:16:33.432754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432572 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.432754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432580 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432587 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:33.432754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432596 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b242b9-5314-48b2-ac7b-9f2e28f393ca" containerName="main" Oct 30 01:16:33.432754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.432603 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:33.437847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.437828 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:33.454893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.454828 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-e71e23"] Oct 30 01:16:33.460844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.460821 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:16:33.560820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.560784 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4q5\" (UniqueName: \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\") pod \"test-curl-8753-e71e23\" (UID: \"919e526c-84c4-4809-8c05-f8e8a3911819\") " pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:33.662686 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.662654 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4q5\" (UniqueName: \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\") pod \"test-curl-8753-e71e23\" (UID: \"919e526c-84c4-4809-8c05-f8e8a3911819\") " pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:33.681352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.681320 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4q5\" (UniqueName: \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\") pod \"test-curl-8753-e71e23\" (UID: \"919e526c-84c4-4809-8c05-f8e8a3911819\") " pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:33.751586 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:33.751518 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:34.056713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:34.056685 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8753-e71e23"] Oct 30 01:16:34.057511 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:16:34.057486 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919e526c_84c4_4809_8c05_f8e8a3911819.slice/crio-40c20f0f69801dc7a7110d2458d624df4026238cc9e009ceee94a4421c5eccbb WatchSource:0}: Error finding container 40c20f0f69801dc7a7110d2458d624df4026238cc9e009ceee94a4421c5eccbb: Status 404 returned error can't find the container with id 40c20f0f69801dc7a7110d2458d624df4026238cc9e009ceee94a4421c5eccbb Oct 30 01:16:34.422917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:34.422885 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-e71e23" event={"ID":"919e526c-84c4-4809-8c05-f8e8a3911819","Type":"ContainerStarted","Data":"211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26"} Oct 30 01:16:34.422917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:34.422921 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8753-e71e23" event={"ID":"919e526c-84c4-4809-8c05-f8e8a3911819","Type":"ContainerStarted","Data":"40c20f0f69801dc7a7110d2458d624df4026238cc9e009ceee94a4421c5eccbb"} Oct 30 01:16:34.459078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:34.459026 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8753-e71e23" podStartSLOduration=1.459009889 podStartE2EDuration="1.459009889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:34.455200774 +0000 UTC m=+1985.297761366" watchObservedRunningTime="2026-10-30 01:16:34.459009889 +0000 UTC m=+1985.301570482" Oct 30 01:16:35.311489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:35.311455 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8753-e71e23"] Oct 30 01:16:35.315697 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:35.315670 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8753-e71e23"] Oct 30 01:16:36.438179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:36.438143 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8753-e71e23" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" containerName="test-curl-8753-e71e23" containerID="cri-o://211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26" gracePeriod=2 Oct 30 01:16:36.449352 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:16:36.449322 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8753-e71e23\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:38.665866 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:38.665839 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:38.740505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:38.740426 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\" (UniqueName: \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\") pod \"919e526c-84c4-4809-8c05-f8e8a3911819\" (UID: \"919e526c-84c4-4809-8c05-f8e8a3911819\") " Oct 30 01:16:38.742693 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:38.742670 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5" pod "919e526c-84c4-4809-8c05-f8e8a3911819" (UID: "919e526c-84c4-4809-8c05-f8e8a3911819"). InnerVolumeSpecName "kube-api-access-gp4q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:38.842520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:38.842493 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4q5\" (UniqueName: \"kubernetes.io/projected/919e526c-84c4-4809-8c05-f8e8a3911819-kube-api-access-gp4q5\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:16:39.265658 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.265629 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" path="/var/lib/kubelet/pods/919e526c-84c4-4809-8c05-f8e8a3911819/volumes" Oct 30 01:16:39.463804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.463767 5548 generic.go:356] "Generic (PLEG): container finished" podID="919e526c-84c4-4809-8c05-f8e8a3911819" containerID="211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26" exitCode=137 Oct 30 01:16:39.464023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.463819 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8753-e71e23" Oct 30 01:16:39.464023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.463879 5548 scope.go:122] "RemoveContainer" containerID="211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26" Oct 30 01:16:39.474801 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.474777 5548 scope.go:122] "RemoveContainer" containerID="211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26" Oct 30 01:16:39.475167 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:16:39.475140 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26\": container with ID starting with 211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26 not found: ID does not exist" containerID="211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26" Oct 30 01:16:39.475270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:39.475176 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26"} err="failed to get container status \"211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26\": rpc error: code = NotFound desc = could not find container \"211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26\": container with ID starting with 211445fc1f72859ca667ea3e01bb5e2f5ccf1d7829f272551185e2d3b874de26 not found: ID does not exist" Oct 30 01:16:47.647849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.647769 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:16:47.648497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648480 5548 cpu_manager.go:414] "containerMap: removing container" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" containerName="test-curl-8753-e71e23" Oct 30 01:16:47.648561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648498 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" containerName="test-curl-8753-e71e23" Oct 30 01:16:47.648561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648529 5548 cpu_manager.go:414] "containerMap: removing container" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:47.648561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648534 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:47.648662 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648648 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="919e526c-84c4-4809-8c05-f8e8a3911819" containerName="test-curl-8753-e71e23" Oct 30 01:16:47.648662 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648658 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dbbe9d-f2c7-4a8a-8096-03c3c43540f6" containerName="cleanup" Oct 30 01:16:47.648726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.648666 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="82daacfd-5e9b-4f41-847d-d7e7c3fa7db6" containerName="cleanup" Oct 30 01:16:47.653785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.653765 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.677623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.677591 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e5711-openshift-default-dockercfg-d87hb\"" Oct 30 01:16:47.680306 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.680281 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:16:47.778072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778037 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778075 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9wgq\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778201 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778244 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778375 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778331 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778373 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778402 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778506 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.778578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.778556 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880448 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880493 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9wgq\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880587 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880648 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880703 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880742 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880786 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880828 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.880979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.880894 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.881196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.881106 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.881196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.881139 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.881375 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.881354 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.881474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.881395 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.881474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.881458 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.883252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.883230 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.883660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.883642 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.899491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.899435 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.899582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.899496 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9wgq\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:47.969821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:47.969781 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:48.302733 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.302706 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:16:48.304390 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:16:48.304360 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98deaedf_94ab_4028_b5f0_81862d70407d.slice/crio-fae2868139b3aeee259b916652705bab5cb48bfd7701036beaa9260094aede36 WatchSource:0}: Error finding container fae2868139b3aeee259b916652705bab5cb48bfd7701036beaa9260094aede36: Status 404 returned error can't find the container with id fae2868139b3aeee259b916652705bab5cb48bfd7701036beaa9260094aede36 Oct 30 01:16:48.307138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.307105 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:16:48.307205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.307172 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:16:48.307205 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.307201 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:16:48.542875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.542811 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" event={"ID":"98deaedf-94ab-4028-b5f0-81862d70407d","Type":"ContainerStarted","Data":"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29"} Oct 30 01:16:48.543080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.542926 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" event={"ID":"98deaedf-94ab-4028-b5f0-81862d70407d","Type":"ContainerStarted","Data":"fae2868139b3aeee259b916652705bab5cb48bfd7701036beaa9260094aede36"} Oct 30 01:16:48.597389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.597288 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podStartSLOduration=1.597273945 podStartE2EDuration="1.597273945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:48.593092071 +0000 UTC m=+1999.435652682" watchObservedRunningTime="2026-10-30 01:16:48.597273945 +0000 UTC m=+1999.439834538" Oct 30 01:16:48.974888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:48.972972 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:49.992077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:49.992037 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:16:49.992610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:49.992120 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:16:50.990885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:50.990256 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:16:50.990885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:50.990334 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:16:51.984944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:51.971281 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:16:51.984944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:51.971362 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:16:52.981884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:52.973237 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:16:52.981884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:52.973297 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:16:53.978890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:53.978260 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:16:53.978890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:53.978322 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 01:16:55.023467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:55.023434 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:55.024405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:55.024380 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:55.025062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:16:55.025040 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:16:55.428514 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:16:55.428478 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:17:00.496690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:00.496654 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:17:15.520123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.520074 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" podUID="c1599b3b-b92e-4656-9241-68148ff64461" containerName="main" containerID="cri-o://7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2" gracePeriod=45 Oct 30 01:17:15.812957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.812928 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:17:15.893305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893272 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.893511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893353 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.893511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893446 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.893649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893632 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\" (UniqueName: \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.893726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893698 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.893726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893709 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.893841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893663 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.893984 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.893935 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\") pod \"c1599b3b-b92e-4656-9241-68148ff64461\" (UID: \"c1599b3b-b92e-4656-9241-68148ff64461\") " Oct 30 01:17:15.894050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.894016 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.894837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.894811 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.895021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.894842 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.895021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.894876 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.896263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.896229 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "kube-api-access-f9fh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:15.896425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.896401 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:15.896549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.896524 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs" pod "c1599b3b-b92e-4656-9241-68148ff64461" (UID: "c1599b3b-b92e-4656-9241-68148ff64461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:15.916134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.916110 5548 generic.go:356] "Generic (PLEG): container finished" podID="c1599b3b-b92e-4656-9241-68148ff64461" containerID="7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2" exitCode=0 Oct 30 01:17:15.916241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.916158 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" Oct 30 01:17:15.916303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.916148 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" event={"ID":"c1599b3b-b92e-4656-9241-68148ff64461","Type":"ContainerDied","Data":"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2"} Oct 30 01:17:15.916303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.916284 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" event={"ID":"c1599b3b-b92e-4656-9241-68148ff64461","Type":"ContainerDied","Data":"064859db7774df0d7f3992da9c8ac99bca628b1edf590152c57fe0c675aff26a"} Oct 30 01:17:15.916384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.916308 5548 scope.go:122] "RemoveContainer" containerID="7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2" Oct 30 01:17:15.931082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.931063 5548 scope.go:122] "RemoveContainer" containerID="7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2" Oct 30 01:17:15.931369 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:15.931350 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2\": container with ID starting with 7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2 not found: ID does not exist" containerID="7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2" Oct 30 01:17:15.931415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.931379 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2"} err="failed to get container status \"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2\": rpc error: code = NotFound desc = could not find container \"7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2\": container with ID starting with 7191839e99692d38570ea8db17d171240708ccd901d60c4bd5cd081bdc5588f2 not found: ID does not exist" Oct 30 01:17:15.962983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.962953 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:17:15.968028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.968006 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd"] Oct 30 01:17:15.996694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.996674 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1599b3b-b92e-4656-9241-68148ff64461-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.996694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.996695 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1599b3b-b92e-4656-9241-68148ff64461-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:15.996844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:15.996705 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fh5\" (UniqueName: \"kubernetes.io/projected/c1599b3b-b92e-4656-9241-68148ff64461-kube-api-access-f9fh5\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:16.706872 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:16.706823 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-64f472/auth-test-model-64f472-kserve-7c7bffbd45-r4wwd" podUID="c1599b3b-b92e-4656-9241-68148ff64461" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:17:17.338959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:17.338918 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1599b3b-b92e-4656-9241-68148ff64461" path="/var/lib/kubelet/pods/c1599b3b-b92e-4656-9241-68148ff64461/volumes" Oct 30 01:17:21.654577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.654544 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:17:21.655245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.655230 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c1599b3b-b92e-4656-9241-68148ff64461" containerName="main" Oct 30 01:17:21.655305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.655247 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1599b3b-b92e-4656-9241-68148ff64461" containerName="main" Oct 30 01:17:21.655407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.655395 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1599b3b-b92e-4656-9241-68148ff64461" containerName="main" Oct 30 01:17:21.660983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.660963 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.693882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.680104 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:17:21.701881 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.699817 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e69dee-openshift-default-dockercfg-lxjwb\"" Oct 30 01:17:21.823038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823005 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823040 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823146 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823197 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xchm\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823437 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823533 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823570 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823656 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823605 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.823710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.823681 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.924947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.924849 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.924952 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.924996 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925045 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925079 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925130 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925155 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925234 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925283 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xchm\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925601 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925663 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925690 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.925919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.925844 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.926073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.926048 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.928067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.928043 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.928261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.928236 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.942325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.942296 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.942549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.942525 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xchm\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:21.976105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:21.976083 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:22.294059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.294032 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:17:22.295309 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:17:22.295282 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c96356_6972_4806_867a_dcce5a895fa3.slice/crio-7ced21c0be3df7c98882c9869b22f42fb46f265830f55ab8e42b1d0f847d04d4 WatchSource:0}: Error finding container 7ced21c0be3df7c98882c9869b22f42fb46f265830f55ab8e42b1d0f847d04d4: Status 404 returned error can't find the container with id 7ced21c0be3df7c98882c9869b22f42fb46f265830f55ab8e42b1d0f847d04d4 Oct 30 01:17:22.298221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.298185 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:17:22.298319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.298254 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:17:22.298319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.298294 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:17:22.970269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.970237 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" event={"ID":"95c96356-6972-4806-867a-dcce5a895fa3","Type":"ContainerStarted","Data":"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965"} Oct 30 01:17:22.970762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.970292 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" event={"ID":"95c96356-6972-4806-867a-dcce5a895fa3","Type":"ContainerStarted","Data":"7ced21c0be3df7c98882c9869b22f42fb46f265830f55ab8e42b1d0f847d04d4"} Oct 30 01:17:22.976941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:22.976597 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:23.589388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.574056 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podStartSLOduration=2.5740337159999997 podStartE2EDuration="2.574033716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:23.02771917 +0000 UTC m=+2033.870279763" watchObservedRunningTime="2026-10-30 01:17:23.574033716 +0000 UTC m=+2034.416594324" Oct 30 01:17:23.589388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.588757 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:17:23.589388 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.589030 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" containerID="cri-o://ae016cb73d940802abfe5a989006f9d4faeea3253d6db6ffff338e8c1bb067c4" gracePeriod=30 Oct 30 01:17:23.795835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.794917 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:17:23.795835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.795544 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" containerID="cri-o://43c727224ac96aacdbf838a8c69786503e8297f635178e0349c8b832737adf4c" gracePeriod=30 Oct 30 01:17:23.999887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.997717 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:23.999887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:23.997786 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:24.991933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:24.979229 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:24.991933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:24.979288 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:25.428985 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:25.428937 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:17:25.908060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:25.908029 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 01:17:25.908316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:25.908288 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" containerID="cri-o://90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695" gracePeriod=30 Oct 30 01:17:25.982333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:25.982295 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:25.982583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:25.982555 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:27.011415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:27.011377 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:27.012047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:27.012013 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:27.988885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:27.988283 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:27.988885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:27.988361 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:28.977289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:28.977256 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 01:17:28.977838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:28.977794 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 01:17:29.336938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.332100 5548 generic.go:356] "Generic (PLEG): container finished" podID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerID="43c727224ac96aacdbf838a8c69786503e8297f635178e0349c8b832737adf4c" exitCode=0 Oct 30 01:17:29.553919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.545843 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:17:29.564426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.564393 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" event={"ID":"80783643-0bcf-4a15-9da9-90e679b2cd60","Type":"ContainerDied","Data":"43c727224ac96aacdbf838a8c69786503e8297f635178e0349c8b832737adf4c"} Oct 30 01:17:29.564820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.564794 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" podUID="53391acd-093d-424a-9829-af156e13d521" containerName="maas-api" containerID="cri-o://4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f" gracePeriod=30 Oct 30 01:17:29.635006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.623714 5548 generic.go:356] "Generic (PLEG): container finished" podID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerID="ae016cb73d940802abfe5a989006f9d4faeea3253d6db6ffff338e8c1bb067c4" exitCode=0 Oct 30 01:17:29.635006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:29.623847 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" event={"ID":"dc8a07ae-22a0-4165-af30-b9c46da08a2e","Type":"ContainerDied","Data":"ae016cb73d940802abfe5a989006f9d4faeea3253d6db6ffff338e8c1bb067c4"} Oct 30 01:17:30.158596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.150044 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:30.158596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.151804 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:30.162798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.162197 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:17:30.306351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.306313 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:17:30.381129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.381095 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:17:30.383457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.383435 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.383928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.383906 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.384351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.384334 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.384602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.384583 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.384776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.384763 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.385172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.385157 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.385550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.385409 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.385550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.385529 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.385670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.385656 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.385753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.385744 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.386108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.386010 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.386263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.386253 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.386711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.386699 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\") pod \"80783643-0bcf-4a15-9da9-90e679b2cd60\" (UID: \"80783643-0bcf-4a15-9da9-90e679b2cd60\") " Oct 30 01:17:30.387623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.387609 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.387736 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.387727 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.387808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.387801 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.388001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.387930 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.388001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.386540 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:30.388001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.386653 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:30.388600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.388568 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.391437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.391402 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "kube-api-access-x7sd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:30.393133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.393102 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo" pod "80783643-0bcf-4a15-9da9-90e679b2cd60" (UID: "80783643-0bcf-4a15-9da9-90e679b2cd60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:30.457685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.457613 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:17:30.488511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488469 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488529 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488583 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488654 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488704 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488785 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488814 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.488916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488842 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.488916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488911 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.489140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.488957 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.489140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489052 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\") pod \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\" (UID: \"dc8a07ae-22a0-4165-af30-b9c46da08a2e\") " Oct 30 01:17:30.489140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489104 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:30.489301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489177 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.489361 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489337 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.489815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489774 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.489815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489795 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.489815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489810 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80783643-0bcf-4a15-9da9-90e679b2cd60-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489827 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489841 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489854 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489888 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7sd6\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-kube-api-access-x7sd6\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489902 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489918 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.490210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.489932 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80783643-0bcf-4a15-9da9-90e679b2cd60-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.491139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.491109 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:30.491613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.491585 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:30.491833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.491817 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:30.491833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.491821 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss" pod "dc8a07ae-22a0-4165-af30-b9c46da08a2e" (UID: "dc8a07ae-22a0-4165-af30-b9c46da08a2e"). InnerVolumeSpecName "kube-api-access-tdwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:30.590810 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.590778 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\" (UniqueName: \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\") pod \"53391acd-093d-424a-9829-af156e13d521\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " Oct 30 01:17:30.591072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.590922 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\") pod \"53391acd-093d-424a-9829-af156e13d521\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " Oct 30 01:17:30.591072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591050 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\") pod \"53391acd-093d-424a-9829-af156e13d521\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " Oct 30 01:17:30.591165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591083 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\") pod \"53391acd-093d-424a-9829-af156e13d521\" (UID: \"53391acd-093d-424a-9829-af156e13d521\") " Oct 30 01:17:30.591454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591401 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca" pod "53391acd-093d-424a-9829-af156e13d521" (UID: "53391acd-093d-424a-9829-af156e13d521"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:30.591602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591582 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.591602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591601 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.591711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591615 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53391acd-093d-424a-9829-af156e13d521-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.591711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591628 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwss\" (UniqueName: \"kubernetes.io/projected/dc8a07ae-22a0-4165-af30-b9c46da08a2e-kube-api-access-tdwss\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.591711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.591642 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc8a07ae-22a0-4165-af30-b9c46da08a2e-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.593203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.593172 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm" pod "53391acd-093d-424a-9829-af156e13d521" (UID: "53391acd-093d-424a-9829-af156e13d521"). InnerVolumeSpecName "kube-api-access-mm4xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:30.593203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.593194 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls" pod "53391acd-093d-424a-9829-af156e13d521" (UID: "53391acd-093d-424a-9829-af156e13d521"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:30.593360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.593331 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs" pod "53391acd-093d-424a-9829-af156e13d521" (UID: "53391acd-093d-424a-9829-af156e13d521"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:30.633913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.633841 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" Oct 30 01:17:30.633913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.633888 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh" event={"ID":"80783643-0bcf-4a15-9da9-90e679b2cd60","Type":"ContainerDied","Data":"65e97a2e9ec41ceb9d46dc8b5b6da1ff3dee4bb33baef421c70b615673465213"} Oct 30 01:17:30.634173 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.633956 5548 scope.go:122] "RemoveContainer" containerID="43c727224ac96aacdbf838a8c69786503e8297f635178e0349c8b832737adf4c" Oct 30 01:17:30.635939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.635909 5548 generic.go:356] "Generic (PLEG): container finished" podID="53391acd-093d-424a-9829-af156e13d521" containerID="4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f" exitCode=0 Oct 30 01:17:30.636052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.636020 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" event={"ID":"53391acd-093d-424a-9829-af156e13d521","Type":"ContainerDied","Data":"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f"} Oct 30 01:17:30.636052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.636037 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" Oct 30 01:17:30.636188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.636063 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq" event={"ID":"53391acd-093d-424a-9829-af156e13d521","Type":"ContainerDied","Data":"03766461b5bd857b74c0beea33ab529a35c4f53dceaf09524c48a9a302b54c81"} Oct 30 01:17:30.638492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.638477 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" Oct 30 01:17:30.638728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.638506 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws" event={"ID":"dc8a07ae-22a0-4165-af30-b9c46da08a2e","Type":"ContainerDied","Data":"e357756136c40c1fe0292b9e01f37c3ca4aab48950a32604d5b69ebf6568fa01"} Oct 30 01:17:30.648134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.648114 5548 scope.go:122] "RemoveContainer" containerID="4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f" Oct 30 01:17:30.687341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.687319 5548 scope.go:122] "RemoveContainer" containerID="4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f" Oct 30 01:17:30.693118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.693090 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4xm\" (UniqueName: \"kubernetes.io/projected/53391acd-093d-424a-9829-af156e13d521-kube-api-access-mm4xm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.693262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.693248 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.693367 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.693356 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53391acd-093d-424a-9829-af156e13d521-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.697353 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:30.697295 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f\": container with ID starting with 4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f not found: ID does not exist" containerID="4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f" Oct 30 01:17:30.697353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.697340 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f"} err="failed to get container status \"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f\": rpc error: code = NotFound desc = could not find container \"4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f\": container with ID starting with 4f7c1e5ce1d24d914576083a8d276bdac49a41340386017733fd80dfec46056f not found: ID does not exist" Oct 30 01:17:30.697639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.697367 5548 scope.go:122] "RemoveContainer" containerID="ae016cb73d940802abfe5a989006f9d4faeea3253d6db6ffff338e8c1bb067c4" Oct 30 01:17:30.752594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.752564 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:17:30.761901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.761844 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-6mxws"] Oct 30 01:17:30.817475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.817441 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:17:30.832572 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:30.832490 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-f67mh"] Oct 30 01:17:31.071031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.070934 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:17:31.077358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.077321 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52f1cb-5757b56cbb-sh2rq"] Oct 30 01:17:31.266355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.266321 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53391acd-093d-424a-9829-af156e13d521" path="/var/lib/kubelet/pods/53391acd-093d-424a-9829-af156e13d521/volumes" Oct 30 01:17:31.267052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.267022 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" path="/var/lib/kubelet/pods/80783643-0bcf-4a15-9da9-90e679b2cd60/volumes" Oct 30 01:17:31.267622 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.267594 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" path="/var/lib/kubelet/pods/dc8a07ae-22a0-4165-af30-b9c46da08a2e/volumes" Oct 30 01:17:31.293389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.293369 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 01:17:31.411316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411279 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411405 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411447 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411479 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411528 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411614 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411677 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411716 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.411785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411760 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\") pod \"2e474b8a-3775-4d39-8226-1fef3c8fecde\" (UID: \"2e474b8a-3775-4d39-8226-1fef3c8fecde\") " Oct 30 01:17:31.412041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411776 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.412041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411927 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.412041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.411944 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:31.412041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.412028 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.412765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.412738 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.413192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.413168 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e474b8a-3775-4d39-8226-1fef3c8fecde-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.413294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.413198 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.413294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.413213 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.413294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.413226 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.414192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.414169 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:31.414393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.414371 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "kube-api-access-zlswk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:31.414499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.414474 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.414551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.414509 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token" pod "2e474b8a-3775-4d39-8226-1fef3c8fecde" (UID: "2e474b8a-3775-4d39-8226-1fef3c8fecde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:31.514771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.514687 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlswk\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-kube-api-access-zlswk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.514771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.514719 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.514771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.514729 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.514771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.514737 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e474b8a-3775-4d39-8226-1fef3c8fecde-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.514771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.514745 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e474b8a-3775-4d39-8226-1fef3c8fecde-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.650702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.650669 5548 generic.go:356] "Generic (PLEG): container finished" podID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerID="90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695" exitCode=0 Oct 30 01:17:31.650938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.650728 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" Oct 30 01:17:31.650938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.650756 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" event={"ID":"2e474b8a-3775-4d39-8226-1fef3c8fecde","Type":"ContainerDied","Data":"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695"} Oct 30 01:17:31.650938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.650814 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" event={"ID":"2e474b8a-3775-4d39-8226-1fef3c8fecde","Type":"ContainerDied","Data":"c2a910db0084446178f869cc69a3abcf7bb7cf5016fceb04107115389098a49c"} Oct 30 01:17:31.650938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.650843 5548 scope.go:122] "RemoveContainer" containerID="90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695" Oct 30 01:17:31.667184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.667164 5548 scope.go:122] "RemoveContainer" containerID="90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695" Oct 30 01:17:31.667509 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:31.667489 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695\": container with ID starting with 90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695 not found: ID does not exist" containerID="90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695" Oct 30 01:17:31.667599 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.667515 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695"} err="failed to get container status \"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695\": rpc error: code = NotFound desc = could not find container \"90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695\": container with ID starting with 90d1eaf4edb7a412ac1f032254ac0e4ce3129b85da9eeb7204ed5e99e4163695 not found: ID does not exist" Oct 30 01:17:31.699410 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.699382 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 01:17:31.707681 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:31.707657 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm"] Oct 30 01:17:32.246494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:32.246457 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f6dvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:17:32.246688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:32.246523 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f6dvm" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:17:33.268243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:33.268209 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" path="/var/lib/kubelet/pods/2e474b8a-3775-4d39-8226-1fef3c8fecde/volumes" Oct 30 01:17:36.896651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:36.896608 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:17:36.897192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:36.896909 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" containerID="cri-o://6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1" gracePeriod=30 Oct 30 01:17:42.189183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.189106 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:17:42.281279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281240 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281283 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281377 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281465 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281500 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281548 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281698 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281831 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281794 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.281910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.281827 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\") pod \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\" (UID: \"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd\") " Oct 30 01:17:42.282303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282185 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:42.282491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282245 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.282491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282397 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.283001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282725 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.283001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282754 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.283001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282768 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.283001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282799 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.283001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.282914 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.285580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.285547 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:42.285580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.285552 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "kube-api-access-7rxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:42.285734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.285577 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.285734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.285615 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token" pod "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" (UID: "746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:42.384713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384679 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.384713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384705 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.384713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384716 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxdw\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-kube-api-access-7rxdw\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.384974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384725 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.384974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384734 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.384974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.384741 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.749342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.749310 5548 generic.go:356] "Generic (PLEG): container finished" podID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerID="6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1" exitCode=0 Oct 30 01:17:42.749518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.749367 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" Oct 30 01:17:42.749518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.749400 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerDied","Data":"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1"} Oct 30 01:17:42.749518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.749461 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w" event={"ID":"746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd","Type":"ContainerDied","Data":"ba21097d4172938c3854e4285ade3977d399019744e58febb4aa4bcb6845bc70"} Oct 30 01:17:42.749518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.749491 5548 scope.go:122] "RemoveContainer" containerID="6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1" Oct 30 01:17:42.764148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.764117 5548 scope.go:122] "RemoveContainer" containerID="2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f" Oct 30 01:17:42.774067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.774048 5548 scope.go:122] "RemoveContainer" containerID="6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1" Oct 30 01:17:42.774346 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:42.774327 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1\": container with ID starting with 6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1 not found: ID does not exist" containerID="6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1" Oct 30 01:17:42.774414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.774359 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1"} err="failed to get container status \"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1\": rpc error: code = NotFound desc = could not find container \"6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1\": container with ID starting with 6e9cdb53d07fd33d3662b2b404783c1151d09067d9db0ba89090199a810766e1 not found: ID does not exist" Oct 30 01:17:42.774414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.774383 5548 scope.go:122] "RemoveContainer" containerID="2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f" Oct 30 01:17:42.774652 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:42.774635 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f\": container with ID starting with 2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f not found: ID does not exist" containerID="2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f" Oct 30 01:17:42.774692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.774659 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f"} err="failed to get container status \"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f\": rpc error: code = NotFound desc = could not find container \"2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f\": container with ID starting with 2ee9aae14a64136883690780bfd180ad880db55c2bf8dd1557676e1909c71a5f not found: ID does not exist" Oct 30 01:17:42.798740 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.798712 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:17:42.803370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:42.803349 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52f1cb-openshift-default-79fc8b8c6f-52t9w"] Oct 30 01:17:43.266844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:17:43.266804 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" path="/var/lib/kubelet/pods/746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd/volumes" Oct 30 01:17:55.429888 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:17:55.429808 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:18:20.545686 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:20.545651 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:18:20.546808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:20.546009 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" podUID="489949ee-01ce-436f-b1a9-c9e667e5e350" containerName="maas-api" containerID="cri-o://fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32" gracePeriod=30 Oct 30 01:18:21.018793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.018748 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:18:21.113386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.112651 5548 generic.go:356] "Generic (PLEG): container finished" podID="489949ee-01ce-436f-b1a9-c9e667e5e350" containerID="fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32" exitCode=0 Oct 30 01:18:21.113386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.112753 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" event={"ID":"489949ee-01ce-436f-b1a9-c9e667e5e350","Type":"ContainerDied","Data":"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32"} Oct 30 01:18:21.113386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.112799 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" event={"ID":"489949ee-01ce-436f-b1a9-c9e667e5e350","Type":"ContainerDied","Data":"8b82ed6b281c70ffea2b36c22abf4d1d53b33f87644b56c5d81ea886ccdf74b0"} Oct 30 01:18:21.113386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.112828 5548 scope.go:122] "RemoveContainer" containerID="fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32" Oct 30 01:18:21.113386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.113015 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8" Oct 30 01:18:21.161978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.161948 5548 scope.go:122] "RemoveContainer" containerID="fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32" Oct 30 01:18:21.162555 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:18:21.162530 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32\": container with ID starting with fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32 not found: ID does not exist" containerID="fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32" Oct 30 01:18:21.170065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.169985 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32"} err="failed to get container status \"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32\": rpc error: code = NotFound desc = could not find container \"fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32\": container with ID starting with fa3ffe25d0434637fb27104c7d8e7bb97c6293cfe8fd5c62c512e126fb0efe32 not found: ID does not exist" Oct 30 01:18:21.199184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.199156 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\") pod \"489949ee-01ce-436f-b1a9-c9e667e5e350\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " Oct 30 01:18:21.199491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.199476 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\") pod \"489949ee-01ce-436f-b1a9-c9e667e5e350\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " Oct 30 01:18:21.199669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.199654 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\" (UniqueName: \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\") pod \"489949ee-01ce-436f-b1a9-c9e667e5e350\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " Oct 30 01:18:21.199836 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.199824 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\") pod \"489949ee-01ce-436f-b1a9-c9e667e5e350\" (UID: \"489949ee-01ce-436f-b1a9-c9e667e5e350\") " Oct 30 01:18:21.200310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.200288 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca" pod "489949ee-01ce-436f-b1a9-c9e667e5e350" (UID: "489949ee-01ce-436f-b1a9-c9e667e5e350"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:21.201418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.201236 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489949ee-01ce-436f-b1a9-c9e667e5e350-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:21.212744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.212694 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls" pod "489949ee-01ce-436f-b1a9-c9e667e5e350" (UID: "489949ee-01ce-436f-b1a9-c9e667e5e350"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:21.212744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.212701 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd" pod "489949ee-01ce-436f-b1a9-c9e667e5e350" (UID: "489949ee-01ce-436f-b1a9-c9e667e5e350"). InnerVolumeSpecName "kube-api-access-w7mjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:21.213688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.213226 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs" pod "489949ee-01ce-436f-b1a9-c9e667e5e350" (UID: "489949ee-01ce-436f-b1a9-c9e667e5e350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:21.304050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.304020 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mjd\" (UniqueName: \"kubernetes.io/projected/489949ee-01ce-436f-b1a9-c9e667e5e350-kube-api-access-w7mjd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:21.304050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.304052 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:21.304341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.304067 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489949ee-01ce-436f-b1a9-c9e667e5e350-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:21.465688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.465601 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:18:21.473618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:21.473587 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-687471-7d96ffc55-sqvn8"] Oct 30 01:18:23.266082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:23.266047 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489949ee-01ce-436f-b1a9-c9e667e5e350" path="/var/lib/kubelet/pods/489949ee-01ce-436f-b1a9-c9e667e5e350/volumes" Oct 30 01:18:25.488364 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:18:25.488321 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:18:25.937853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:25.937811 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:18:25.938152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:25.938110 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" containerID="cri-o://2c7c997d5c8b0a23fc0682c2fdabc359aa26dcc94c44dd03cb632466dcae338a" gracePeriod=30 Oct 30 01:18:29.358104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:29.358070 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:18:29.358939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:29.358901 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" containerID="cri-o://e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4" gracePeriod=30 Oct 30 01:18:29.671025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:29.670995 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:18:29.675467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:29.675444 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:18:31.219018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.218743 5548 generic.go:356] "Generic (PLEG): container finished" podID="af382efd-8ef0-44fc-a504-b667e138ed43" containerID="2c7c997d5c8b0a23fc0682c2fdabc359aa26dcc94c44dd03cb632466dcae338a" exitCode=0 Oct 30 01:18:31.219018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.218797 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerDied","Data":"2c7c997d5c8b0a23fc0682c2fdabc359aa26dcc94c44dd03cb632466dcae338a"} Oct 30 01:18:31.219018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.218903 5548 scope.go:122] "RemoveContainer" containerID="1668f8e1d4fbc7f9591a800e25f332578b2db145a1b466bec5eac7d61219a13b" Oct 30 01:18:31.285905 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.285880 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:18:31.405638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405606 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.405834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405649 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.405834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405675 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.405834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405747 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.405834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405802 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405836 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405876 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405906 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.405945 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\") pod \"af382efd-8ef0-44fc-a504-b667e138ed43\" (UID: \"af382efd-8ef0-44fc-a504-b667e138ed43\") " Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406022 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:31.406144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406138 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:31.406458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406347 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:31.406760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406732 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:31.406880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406838 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.407015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406891 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.407015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.406908 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.407198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.407041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:31.408687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.408647 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "kube-api-access-w9rhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:31.409105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.409082 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:31.409551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.409533 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:18:31.410100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.410071 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token" pod "af382efd-8ef0-44fc-a504-b667e138ed43" (UID: "af382efd-8ef0-44fc-a504-b667e138ed43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:31.508026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.507989 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af382efd-8ef0-44fc-a504-b667e138ed43-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.508026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.508021 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af382efd-8ef0-44fc-a504-b667e138ed43-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.508026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.508032 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.508283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.508042 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.508283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.508054 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af382efd-8ef0-44fc-a504-b667e138ed43-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:31.508283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:31.508065 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rhh\" (UniqueName: \"kubernetes.io/projected/af382efd-8ef0-44fc-a504-b667e138ed43-kube-api-access-w9rhh\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:32.230394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:32.230366 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" Oct 30 01:18:32.230937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:32.230393 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72" event={"ID":"af382efd-8ef0-44fc-a504-b667e138ed43","Type":"ContainerDied","Data":"da7658be6838514ccd27a1ab33d329a3d8dda06e119f6699b59461815aa95287"} Oct 30 01:18:32.230937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:32.230466 5548 scope.go:122] "RemoveContainer" containerID="2c7c997d5c8b0a23fc0682c2fdabc359aa26dcc94c44dd03cb632466dcae338a" Oct 30 01:18:32.279772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:32.279738 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:18:32.287339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:32.287303 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e92dab-openshift-default-55f9bdc949-fdg72"] Oct 30 01:18:33.266360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:33.266328 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" path="/var/lib/kubelet/pods/af382efd-8ef0-44fc-a504-b667e138ed43/volumes" Oct 30 01:18:34.672928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.672900 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:18:34.759929 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.759893 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760202 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760176 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760248 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760320 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760393 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760434 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760532 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:34.760604 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760541 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760616 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760737 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\") pod \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\" (UID: \"b0c2373c-0ab6-4b45-b34d-39165dc21d94\") " Oct 30 01:18:34.760769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.760740 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:34.761348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761081 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:34.761348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761173 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:34.761348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761165 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:34.761701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761442 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.761701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761464 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.761701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761478 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.761701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761491 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.761701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.761505 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.763301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.763246 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "kube-api-access-tl2g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:34.763825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.763789 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:18:34.763825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.763796 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:34.764768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.764739 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token" pod "b0c2373c-0ab6-4b45-b34d-39165dc21d94" (UID: "b0c2373c-0ab6-4b45-b34d-39165dc21d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:34.862468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.862433 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.862468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.862469 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2g4\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-kube-api-access-tl2g4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.862679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.862483 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:34.862679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:34.862495 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c2373c-0ab6-4b45-b34d-39165dc21d94-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:18:35.255381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.255341 5548 generic.go:356] "Generic (PLEG): container finished" podID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerID="e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4" exitCode=0 Oct 30 01:18:35.255577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.255406 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" Oct 30 01:18:35.255577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.255417 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" event={"ID":"b0c2373c-0ab6-4b45-b34d-39165dc21d94","Type":"ContainerDied","Data":"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4"} Oct 30 01:18:35.255577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.255500 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8" event={"ID":"b0c2373c-0ab6-4b45-b34d-39165dc21d94","Type":"ContainerDied","Data":"7403f1006250f1840e0c64ef3b08b0f8781444ffe93d62828cb8ab2f915724ac"} Oct 30 01:18:35.255577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.255523 5548 scope.go:122] "RemoveContainer" containerID="e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4" Oct 30 01:18:35.267650 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.267628 5548 scope.go:122] "RemoveContainer" containerID="e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4" Oct 30 01:18:35.267964 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:18:35.267941 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4\": container with ID starting with e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4 not found: ID does not exist" containerID="e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4" Oct 30 01:18:35.268052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.267976 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4"} err="failed to get container status \"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4\": rpc error: code = NotFound desc = could not find container \"e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4\": container with ID starting with e30948f979f012cb4851b1e9f84f86d60ae45050a48821376baff5866c53b2d4 not found: ID does not exist" Oct 30 01:18:35.312371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.312344 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:18:35.322585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:35.322552 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-687471-openshift-default-79cdb54bc5-ggsx8"] Oct 30 01:18:37.266128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:18:37.266088 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" path="/var/lib/kubelet/pods/b0c2373c-0ab6-4b45-b34d-39165dc21d94/volumes" Oct 30 01:18:55.441905 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:18:55.441840 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:19:00.003424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.003390 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:19:00.003883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.003604 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" podUID="491e2c6e-3796-45d0-9746-36c11b02482b" containerName="maas-api" containerID="cri-o://9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a" gracePeriod=30 Oct 30 01:19:00.450090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.447941 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:19:00.450090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.448967 5548 generic.go:356] "Generic (PLEG): container finished" podID="491e2c6e-3796-45d0-9746-36c11b02482b" containerID="9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a" exitCode=0 Oct 30 01:19:00.450090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.449040 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" event={"ID":"491e2c6e-3796-45d0-9746-36c11b02482b","Type":"ContainerDied","Data":"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a"} Oct 30 01:19:00.450090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.449090 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" event={"ID":"491e2c6e-3796-45d0-9746-36c11b02482b","Type":"ContainerDied","Data":"dd729c3920443e4b329f291ac2709be6a0828502b98f3a2923f4f154e10dc9d8"} Oct 30 01:19:00.450090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.449118 5548 scope.go:122] "RemoveContainer" containerID="9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a" Oct 30 01:19:00.467314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.467285 5548 scope.go:122] "RemoveContainer" containerID="9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a" Oct 30 01:19:00.467749 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:00.467724 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a\": container with ID starting with 9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a not found: ID does not exist" containerID="9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a" Oct 30 01:19:00.467847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.467761 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a"} err="failed to get container status \"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a\": rpc error: code = NotFound desc = could not find container \"9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a\": container with ID starting with 9f5a1c4d1343cc9036e981d4114ea476124e1db52024d5db386c3a3f1f25087a not found: ID does not exist" Oct 30 01:19:00.569586 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.569534 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\") pod \"491e2c6e-3796-45d0-9746-36c11b02482b\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " Oct 30 01:19:00.569780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.569763 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") pod \"491e2c6e-3796-45d0-9746-36c11b02482b\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " Oct 30 01:19:00.569931 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.569914 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\") pod \"491e2c6e-3796-45d0-9746-36c11b02482b\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " Oct 30 01:19:00.570004 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.569961 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\" (UniqueName: \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\") pod \"491e2c6e-3796-45d0-9746-36c11b02482b\" (UID: \"491e2c6e-3796-45d0-9746-36c11b02482b\") " Oct 30 01:19:00.571262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.571189 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca" pod "491e2c6e-3796-45d0-9746-36c11b02482b" (UID: "491e2c6e-3796-45d0-9746-36c11b02482b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:00.572274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.572251 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls" pod "491e2c6e-3796-45d0-9746-36c11b02482b" (UID: "491e2c6e-3796-45d0-9746-36c11b02482b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:00.572931 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.572907 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8" pod "491e2c6e-3796-45d0-9746-36c11b02482b" (UID: "491e2c6e-3796-45d0-9746-36c11b02482b"). InnerVolumeSpecName "kube-api-access-zwpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:00.573500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.573480 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs" pod "491e2c6e-3796-45d0-9746-36c11b02482b" (UID: "491e2c6e-3796-45d0-9746-36c11b02482b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:00.671757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.671720 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/491e2c6e-3796-45d0-9746-36c11b02482b-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:00.671757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.671759 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpv8\" (UniqueName: \"kubernetes.io/projected/491e2c6e-3796-45d0-9746-36c11b02482b-kube-api-access-zwpv8\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:00.672038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.671773 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:00.672038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:00.671782 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/491e2c6e-3796-45d0-9746-36c11b02482b-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:01.456446 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:01.456415 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t" Oct 30 01:19:01.529371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:01.529284 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:19:01.548977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:01.548945 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-82e9bf-84cd985545-8j76t"] Oct 30 01:19:03.275639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:03.275602 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491e2c6e-3796-45d0-9746-36c11b02482b" path="/var/lib/kubelet/pods/491e2c6e-3796-45d0-9746-36c11b02482b/volumes" Oct 30 01:19:04.239825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.239784 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:19:04.240086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.240051 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" podUID="adec1f7e-c0ee-450a-a896-c935aab0a77a" containerName="maas-api" containerID="cri-o://4702f1c44795610ffde502d92e3ee37ae1f25755c011238e127d7669142af146" gracePeriod=30 Oct 30 01:19:04.602454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.602411 5548 generic.go:356] "Generic (PLEG): container finished" podID="adec1f7e-c0ee-450a-a896-c935aab0a77a" containerID="4702f1c44795610ffde502d92e3ee37ae1f25755c011238e127d7669142af146" exitCode=0 Oct 30 01:19:04.602970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.602671 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" event={"ID":"adec1f7e-c0ee-450a-a896-c935aab0a77a","Type":"ContainerDied","Data":"4702f1c44795610ffde502d92e3ee37ae1f25755c011238e127d7669142af146"} Oct 30 01:19:04.633413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.631637 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:19:04.790395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.790357 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\" (UniqueName: \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\") pod \"adec1f7e-c0ee-450a-a896-c935aab0a77a\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " Oct 30 01:19:04.790587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.790484 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") pod \"adec1f7e-c0ee-450a-a896-c935aab0a77a\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " Oct 30 01:19:04.790587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.790535 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\") pod \"adec1f7e-c0ee-450a-a896-c935aab0a77a\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " Oct 30 01:19:04.790587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.790575 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\") pod \"adec1f7e-c0ee-450a-a896-c935aab0a77a\" (UID: \"adec1f7e-c0ee-450a-a896-c935aab0a77a\") " Oct 30 01:19:04.791605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.791573 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca" pod "adec1f7e-c0ee-450a-a896-c935aab0a77a" (UID: "adec1f7e-c0ee-450a-a896-c935aab0a77a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:04.801156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.801120 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb" pod "adec1f7e-c0ee-450a-a896-c935aab0a77a" (UID: "adec1f7e-c0ee-450a-a896-c935aab0a77a"). InnerVolumeSpecName "kube-api-access-d77tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:04.801307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.801278 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs" pod "adec1f7e-c0ee-450a-a896-c935aab0a77a" (UID: "adec1f7e-c0ee-450a-a896-c935aab0a77a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:04.801593 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.801566 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls" pod "adec1f7e-c0ee-450a-a896-c935aab0a77a" (UID: "adec1f7e-c0ee-450a-a896-c935aab0a77a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:04.892760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.892732 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77tb\" (UniqueName: \"kubernetes.io/projected/adec1f7e-c0ee-450a-a896-c935aab0a77a-kube-api-access-d77tb\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:04.892962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.892769 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:04.892962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.892785 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adec1f7e-c0ee-450a-a896-c935aab0a77a-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:04.892962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:04.892803 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adec1f7e-c0ee-450a-a896-c935aab0a77a-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:05.610313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:05.610231 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" Oct 30 01:19:05.610782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:05.610330 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n" event={"ID":"adec1f7e-c0ee-450a-a896-c935aab0a77a","Type":"ContainerDied","Data":"6930d480add8c850e6d138841bb602e54bbfdbc0e70f7731e4eb6137549d61f9"} Oct 30 01:19:05.610782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:05.610434 5548 scope.go:122] "RemoveContainer" containerID="4702f1c44795610ffde502d92e3ee37ae1f25755c011238e127d7669142af146" Oct 30 01:19:05.683132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:05.683102 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:19:05.693331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:05.693303 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-56d1d9-6f966697b9-vgn4n"] Oct 30 01:19:06.471062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.471035 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:19:06.471281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.471254 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" podUID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" containerName="maas-api" containerID="cri-o://5c75299519eabebbfbe4993bf3451fa2ab7dd3b808805e806f6846fabcc8b7ff" gracePeriod=30 Oct 30 01:19:06.632544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.632508 5548 generic.go:356] "Generic (PLEG): container finished" podID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" containerID="5c75299519eabebbfbe4993bf3451fa2ab7dd3b808805e806f6846fabcc8b7ff" exitCode=0 Oct 30 01:19:06.632544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.632539 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" event={"ID":"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8","Type":"ContainerDied","Data":"5c75299519eabebbfbe4993bf3451fa2ab7dd3b808805e806f6846fabcc8b7ff"} Oct 30 01:19:06.786753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.786716 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:19:06.924079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.924045 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\") pod \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " Oct 30 01:19:06.924286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.924099 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\") pod \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " Oct 30 01:19:06.924286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.924189 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") pod \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " Oct 30 01:19:06.924286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.924279 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\" (UniqueName: \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\") pod \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\" (UID: \"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8\") " Oct 30 01:19:06.962940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.950577 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr" pod "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8"). InnerVolumeSpecName "kube-api-access-9l2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:06.962940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.951093 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca" pod "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:06.962940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:06.951623 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs" pod "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:07.015995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.014462 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls" pod "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" (UID: "cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:07.032070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.031990 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:07.032070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.032029 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2xr\" (UniqueName: \"kubernetes.io/projected/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-kube-api-access-9l2xr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:07.032070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.032046 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:07.032070 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.032063 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:07.266809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.266733 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adec1f7e-c0ee-450a-a896-c935aab0a77a" path="/var/lib/kubelet/pods/adec1f7e-c0ee-450a-a896-c935aab0a77a/volumes" Oct 30 01:19:07.641120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.641039 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" Oct 30 01:19:07.641120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.641068 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4" event={"ID":"cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8","Type":"ContainerDied","Data":"a2249152167e8cbcce5857c9f458b01f260b203c81b5eb734ac303980f891273"} Oct 30 01:19:07.641608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.641150 5548 scope.go:122] "RemoveContainer" containerID="5c75299519eabebbfbe4993bf3451fa2ab7dd3b808805e806f6846fabcc8b7ff" Oct 30 01:19:07.696983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.696953 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:19:07.705616 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:07.705577 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50d078-854bb9477f-thxh4"] Oct 30 01:19:09.269729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:09.269689 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" path="/var/lib/kubelet/pods/cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8/volumes" Oct 30 01:19:25.429340 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:25.429296 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:19:36.499138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.499104 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:19:36.499670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.499310 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" podUID="3687f115-8b30-4cbe-989e-096d7e06d3a6" containerName="maas-api" containerID="cri-o://6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1" gracePeriod=30 Oct 30 01:19:36.811778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.811750 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:19:36.904483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.902468 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") pod \"3687f115-8b30-4cbe-989e-096d7e06d3a6\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " Oct 30 01:19:36.904483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.902569 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\") pod \"3687f115-8b30-4cbe-989e-096d7e06d3a6\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " Oct 30 01:19:36.904483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.902711 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\") pod \"3687f115-8b30-4cbe-989e-096d7e06d3a6\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " Oct 30 01:19:36.904483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.902752 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\" (UniqueName: \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\") pod \"3687f115-8b30-4cbe-989e-096d7e06d3a6\" (UID: \"3687f115-8b30-4cbe-989e-096d7e06d3a6\") " Oct 30 01:19:36.904483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.904211 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca" pod "3687f115-8b30-4cbe-989e-096d7e06d3a6" (UID: "3687f115-8b30-4cbe-989e-096d7e06d3a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:36.951948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.945095 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2" pod "3687f115-8b30-4cbe-989e-096d7e06d3a6" (UID: "3687f115-8b30-4cbe-989e-096d7e06d3a6"). InnerVolumeSpecName "kube-api-access-szbx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:36.951948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.945972 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs" pod "3687f115-8b30-4cbe-989e-096d7e06d3a6" (UID: "3687f115-8b30-4cbe-989e-096d7e06d3a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:36.968907 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.952499 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls" pod "3687f115-8b30-4cbe-989e-096d7e06d3a6" (UID: "3687f115-8b30-4cbe-989e-096d7e06d3a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:36.974462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.974425 5548 generic.go:356] "Generic (PLEG): container finished" podID="3687f115-8b30-4cbe-989e-096d7e06d3a6" containerID="6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1" exitCode=0 Oct 30 01:19:36.990901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.974756 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" Oct 30 01:19:36.990901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.974779 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" event={"ID":"3687f115-8b30-4cbe-989e-096d7e06d3a6","Type":"ContainerDied","Data":"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1"} Oct 30 01:19:36.990901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.977174 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr" event={"ID":"3687f115-8b30-4cbe-989e-096d7e06d3a6","Type":"ContainerDied","Data":"ea47d203435ef00659ded3b905c83373eac078922993d23d6256d3ff14e34993"} Oct 30 01:19:36.990901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:36.977214 5548 scope.go:122] "RemoveContainer" containerID="6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1" Oct 30 01:19:37.010152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.010063 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.010152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.010092 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3687f115-8b30-4cbe-989e-096d7e06d3a6-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.010152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.010108 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3687f115-8b30-4cbe-989e-096d7e06d3a6-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.010152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.010127 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbx2\" (UniqueName: \"kubernetes.io/projected/3687f115-8b30-4cbe-989e-096d7e06d3a6-kube-api-access-szbx2\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.058092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.043158 5548 scope.go:122] "RemoveContainer" containerID="6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1" Oct 30 01:19:37.058092 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:37.043552 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1\": container with ID starting with 6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1 not found: ID does not exist" containerID="6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1" Oct 30 01:19:37.058092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.043594 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1"} err="failed to get container status \"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1\": rpc error: code = NotFound desc = could not find container \"6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1\": container with ID starting with 6b6e4e0ec6ae42cbbc01ef1cf3346e2d4d327fd89ccac79c09042c925c4296e1 not found: ID does not exist" Oct 30 01:19:37.126342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.126308 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:19:37.129626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.129602 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e69dee-6cd98bcc8b-mvhnr"] Oct 30 01:19:37.266849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:37.266812 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3687f115-8b30-4cbe-989e-096d7e06d3a6" path="/var/lib/kubelet/pods/3687f115-8b30-4cbe-989e-096d7e06d3a6/volumes" Oct 30 01:19:43.777982 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:43.777904 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:19:43.778541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:43.778137 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" podUID="c4e3ac13-083a-4dfa-905e-56c5ae912036" containerName="maas-api" containerID="cri-o://fced27f451aed375a4306b4438efc004f430ce7bca5dcb544e6beaaac5a230b6" gracePeriod=30 Oct 30 01:19:44.034706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.034673 5548 generic.go:356] "Generic (PLEG): container finished" podID="c4e3ac13-083a-4dfa-905e-56c5ae912036" containerID="fced27f451aed375a4306b4438efc004f430ce7bca5dcb544e6beaaac5a230b6" exitCode=0 Oct 30 01:19:44.034846 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.034743 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" event={"ID":"c4e3ac13-083a-4dfa-905e-56c5ae912036","Type":"ContainerDied","Data":"fced27f451aed375a4306b4438efc004f430ce7bca5dcb544e6beaaac5a230b6"} Oct 30 01:19:44.071727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.071696 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:19:44.102824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.102798 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\") pod \"c4e3ac13-083a-4dfa-905e-56c5ae912036\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " Oct 30 01:19:44.103470 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.103448 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") pod \"c4e3ac13-083a-4dfa-905e-56c5ae912036\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " Oct 30 01:19:44.103679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.103662 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\" (UniqueName: \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\") pod \"c4e3ac13-083a-4dfa-905e-56c5ae912036\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " Oct 30 01:19:44.103853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.103332 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca" pod "c4e3ac13-083a-4dfa-905e-56c5ae912036" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:44.104128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.104104 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\") pod \"c4e3ac13-083a-4dfa-905e-56c5ae912036\" (UID: \"c4e3ac13-083a-4dfa-905e-56c5ae912036\") " Oct 30 01:19:44.104684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.104645 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3ac13-083a-4dfa-905e-56c5ae912036-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.106334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.106307 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls" pod "c4e3ac13-083a-4dfa-905e-56c5ae912036" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:44.106565 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.106535 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m" pod "c4e3ac13-083a-4dfa-905e-56c5ae912036" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036"). InnerVolumeSpecName "kube-api-access-qw99m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:44.107288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.107261 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs" pod "c4e3ac13-083a-4dfa-905e-56c5ae912036" (UID: "c4e3ac13-083a-4dfa-905e-56c5ae912036"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:44.206434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.206393 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.206434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.206428 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e3ac13-083a-4dfa-905e-56c5ae912036-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.206434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.206441 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw99m\" (UniqueName: \"kubernetes.io/projected/c4e3ac13-083a-4dfa-905e-56c5ae912036-kube-api-access-qw99m\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.331001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.324808 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:19:44.331001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:44.325206 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" containerID="cri-o://89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965" gracePeriod=30 Oct 30 01:19:45.044431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.044395 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" Oct 30 01:19:45.044898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.044394 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb" event={"ID":"c4e3ac13-083a-4dfa-905e-56c5ae912036","Type":"ContainerDied","Data":"96efc58cf68251abdd93b8c93786831eefa8aa7d91e5dfc137c06f9a54730daa"} Oct 30 01:19:45.044898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.044550 5548 scope.go:122] "RemoveContainer" containerID="fced27f451aed375a4306b4438efc004f430ce7bca5dcb544e6beaaac5a230b6" Oct 30 01:19:45.093579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.093552 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:19:45.103334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.103308 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e5711-7c87bc9c8c-sbpkb"] Oct 30 01:19:45.154104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.154073 5548 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:19:45.154271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.154122 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:19:45.266668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:45.266632 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e3ac13-083a-4dfa-905e-56c5ae912036" path="/var/lib/kubelet/pods/c4e3ac13-083a-4dfa-905e-56c5ae912036/volumes" Oct 30 01:19:49.717034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.716956 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:19:49.840509 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840480 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.840726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840527 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.840726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840576 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.840726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840680 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.840944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840740 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.840944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840833 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.841063 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.840947 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.841126 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841060 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.841189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841139 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\") pod \"95c96356-6972-4806-867a-dcce5a895fa3\" (UID: \"95c96356-6972-4806-867a-dcce5a895fa3\") " Oct 30 01:19:49.842221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841728 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.842221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841749 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.842221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841883 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.842221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.841913 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.842221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.842184 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:49.843449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.843421 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:49.843575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.843452 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "kube-api-access-5xchm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:49.843575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.843486 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.843782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.843762 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo" pod "95c96356-6972-4806-867a-dcce5a895fa3" (UID: "95c96356-6972-4806-867a-dcce5a895fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:49.943322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943292 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xchm\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-kube-api-access-5xchm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943321 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c96356-6972-4806-867a-dcce5a895fa3-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943334 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943341 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c96356-6972-4806-867a-dcce5a895fa3-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943351 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c96356-6972-4806-867a-dcce5a895fa3-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943358 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943366 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943374 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.943556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:49.943382 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c96356-6972-4806-867a-dcce5a895fa3-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:50.112919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.112812 5548 generic.go:356] "Generic (PLEG): container finished" podID="95c96356-6972-4806-867a-dcce5a895fa3" containerID="89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965" exitCode=0 Oct 30 01:19:50.112919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.112888 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" event={"ID":"95c96356-6972-4806-867a-dcce5a895fa3","Type":"ContainerDied","Data":"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965"} Oct 30 01:19:50.113160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.112939 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" event={"ID":"95c96356-6972-4806-867a-dcce5a895fa3","Type":"ContainerDied","Data":"7ced21c0be3df7c98882c9869b22f42fb46f265830f55ab8e42b1d0f847d04d4"} Oct 30 01:19:50.113160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.112967 5548 scope.go:122] "RemoveContainer" containerID="89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965" Oct 30 01:19:50.113160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.113098 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d" Oct 30 01:19:50.132282 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.132262 5548 scope.go:122] "RemoveContainer" containerID="89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965" Oct 30 01:19:50.132629 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:50.132606 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965\": container with ID starting with 89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965 not found: ID does not exist" containerID="89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965" Oct 30 01:19:50.132678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.132640 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965"} err="failed to get container status \"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965\": rpc error: code = NotFound desc = could not find container \"89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965\": container with ID starting with 89ab99ee5eff786a60e2ae295e630686a81bb1728ebdb72af9f3661aea22e965 not found: ID does not exist" Oct 30 01:19:50.169773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.169741 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:19:50.173451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:50.173424 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-tk48d"] Oct 30 01:19:51.266003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:51.265969 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c96356-6972-4806-867a-dcce5a895fa3" path="/var/lib/kubelet/pods/95c96356-6972-4806-867a-dcce5a895fa3/volumes" Oct 30 01:19:52.570473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:52.563160 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:19:52.570473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:52.563394 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" containerID="cri-o://52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29" gracePeriod=30 Oct 30 01:19:55.026040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:55.026002 5548 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:19:55.026597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:55.026076 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:19:55.427620 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:55.427577 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:19:57.844083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.844020 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:19:57.979804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.979767 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.979851 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.979904 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.979941 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980065 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980110 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980147 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980140 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.980425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980208 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.980425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980241 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980280 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\") pod \"98deaedf-94ab-4028-b5f0-81862d70407d\" (UID: \"98deaedf-94ab-4028-b5f0-81862d70407d\") " Oct 30 01:19:57.980425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980306 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.980647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980446 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:57.980647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980618 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.980902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980883 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98deaedf-94ab-4028-b5f0-81862d70407d-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.980998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980908 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.980998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980925 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.980998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980940 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.980998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.980953 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.982602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.982569 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "kube-api-access-h9wgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:57.982718 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.982624 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.982977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.982959 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:57.983061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:57.983031 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token" pod "98deaedf-94ab-4028-b5f0-81862d70407d" (UID: "98deaedf-94ab-4028-b5f0-81862d70407d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:58.081722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.081680 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:58.081722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.081714 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9wgq\" (UniqueName: \"kubernetes.io/projected/98deaedf-94ab-4028-b5f0-81862d70407d-kube-api-access-h9wgq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:58.081722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.081729 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98deaedf-94ab-4028-b5f0-81862d70407d-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:58.081990 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.081741 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98deaedf-94ab-4028-b5f0-81862d70407d-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:19:58.176392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.176361 5548 generic.go:356] "Generic (PLEG): container finished" podID="98deaedf-94ab-4028-b5f0-81862d70407d" containerID="52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29" exitCode=0 Oct 30 01:19:58.176606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.176411 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" Oct 30 01:19:58.176606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.176440 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" event={"ID":"98deaedf-94ab-4028-b5f0-81862d70407d","Type":"ContainerDied","Data":"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29"} Oct 30 01:19:58.176606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.176506 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8" event={"ID":"98deaedf-94ab-4028-b5f0-81862d70407d","Type":"ContainerDied","Data":"fae2868139b3aeee259b916652705bab5cb48bfd7701036beaa9260094aede36"} Oct 30 01:19:58.176606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.176536 5548 scope.go:122] "RemoveContainer" containerID="52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29" Oct 30 01:19:58.187594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.187577 5548 scope.go:122] "RemoveContainer" containerID="52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29" Oct 30 01:19:58.187878 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:19:58.187840 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29\": container with ID starting with 52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29 not found: ID does not exist" containerID="52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29" Oct 30 01:19:58.187986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.187898 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29"} err="failed to get container status \"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29\": rpc error: code = NotFound desc = could not find container \"52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29\": container with ID starting with 52585be7cbe9cd3ef0c04ae3fde3e0d09b1584e95086a648bcb7aad8c4b20d29 not found: ID does not exist" Oct 30 01:19:58.221701 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.221674 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:19:58.232193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:58.232166 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-m6kh8"] Oct 30 01:19:59.264347 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:19:59.264315 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" path="/var/lib/kubelet/pods/98deaedf-94ab-4028-b5f0-81862d70407d/volumes" Oct 30 01:20:10.298632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.298598 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8768-ea9aaa"] Oct 30 01:20:10.299231 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299216 5548 cpu_manager.go:414] "containerMap: removing container" podUID="489949ee-01ce-436f-b1a9-c9e667e5e350" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299232 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489949ee-01ce-436f-b1a9-c9e667e5e350" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299249 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c4e3ac13-083a-4dfa-905e-56c5ae912036" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299258 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e3ac13-083a-4dfa-905e-56c5ae912036" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299272 5548 cpu_manager.go:414] "containerMap: removing container" podUID="53391acd-093d-424a-9829-af156e13d521" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299278 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53391acd-093d-424a-9829-af156e13d521" containerName="maas-api" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299288 5548 cpu_manager.go:414] "containerMap: removing container" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299293 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" Oct 30 01:20:10.299299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299302 5548 cpu_manager.go:414] "containerMap: removing container" podUID="adec1f7e-c0ee-450a-a896-c935aab0a77a" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299307 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adec1f7e-c0ee-450a-a896-c935aab0a77a" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299313 5548 cpu_manager.go:414] "containerMap: removing container" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299317 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299324 5548 cpu_manager.go:414] "containerMap: removing container" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299327 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299337 5548 cpu_manager.go:414] "containerMap: removing container" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299341 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299349 5548 cpu_manager.go:414] "containerMap: removing container" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299353 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299361 5548 cpu_manager.go:414] "containerMap: removing container" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299365 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299372 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3687f115-8b30-4cbe-989e-096d7e06d3a6" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299376 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3687f115-8b30-4cbe-989e-096d7e06d3a6" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299385 5548 cpu_manager.go:414] "containerMap: removing container" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299390 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299402 5548 cpu_manager.go:414] "containerMap: removing container" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299407 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299412 5548 cpu_manager.go:414] "containerMap: removing container" podUID="491e2c6e-3796-45d0-9746-36c11b02482b" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299416 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491e2c6e-3796-45d0-9746-36c11b02482b" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299423 5548 cpu_manager.go:414] "containerMap: removing container" podUID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299428 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299442 5548 cpu_manager.go:414] "containerMap: removing container" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299448 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299459 5548 cpu_manager.go:414] "containerMap: removing container" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299463 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299542 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299549 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="489949ee-01ce-436f-b1a9-c9e667e5e350" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299560 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e3ac13-083a-4dfa-905e-56c5ae912036" containerName="maas-api" Oct 30 01:20:10.299552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299565 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="53391acd-093d-424a-9829-af156e13d521" containerName="maas-api" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299571 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="98deaedf-94ab-4028-b5f0-81862d70407d" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299579 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="adec1f7e-c0ee-450a-a896-c935aab0a77a" containerName="maas-api" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299584 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c2373c-0ab6-4b45-b34d-39165dc21d94" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299590 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299596 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="80783643-0bcf-4a15-9da9-90e679b2cd60" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299602 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="746e4137-cbc7-44a4-a0c8-d22ab1ca2cfd" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299611 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8a07ae-22a0-4165-af30-b9c46da08a2e" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299620 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3687f115-8b30-4cbe-989e-096d7e06d3a6" containerName="maas-api" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299625 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c96356-6972-4806-867a-dcce5a895fa3" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299631 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="af382efd-8ef0-44fc-a504-b667e138ed43" containerName="istio-proxy" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299636 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="491e2c6e-3796-45d0-9746-36c11b02482b" containerName="maas-api" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299642 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb32c2a6-8ddc-48ea-a34d-5083ea8e5cf8" containerName="maas-api" Oct 30 01:20:10.300507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.299647 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e474b8a-3775-4d39-8226-1fef3c8fecde" containerName="istio-proxy" Oct 30 01:20:10.304639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.304610 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:10.318044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.318021 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-ea9aaa"] Oct 30 01:20:10.323525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.323498 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gkw58\"" Oct 30 01:20:10.341536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.341504 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\") pod \"test-curl-8768-ea9aaa\" (UID: \"49b44719-fbc8-4fd5-9a70-b45ff547c813\") " pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:10.443015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.442984 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\") pod \"test-curl-8768-ea9aaa\" (UID: \"49b44719-fbc8-4fd5-9a70-b45ff547c813\") " pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:10.460277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.460238 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\") pod \"test-curl-8768-ea9aaa\" (UID: \"49b44719-fbc8-4fd5-9a70-b45ff547c813\") " pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:10.617893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.617795 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:10.919111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:10.919079 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-ea9aaa"] Oct 30 01:20:10.920560 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:20:10.920523 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49b44719_fbc8_4fd5_9a70_b45ff547c813.slice/crio-1c183fe49df1eaac6d8efdc5deb54da489bb152460d03a4c6daa339d98704b78 WatchSource:0}: Error finding container 1c183fe49df1eaac6d8efdc5deb54da489bb152460d03a4c6daa339d98704b78: Status 404 returned error can't find the container with id 1c183fe49df1eaac6d8efdc5deb54da489bb152460d03a4c6daa339d98704b78 Oct 30 01:20:11.289083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:11.288990 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-ea9aaa" event={"ID":"49b44719-fbc8-4fd5-9a70-b45ff547c813","Type":"ContainerStarted","Data":"e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4"} Oct 30 01:20:11.289083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:11.289025 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-ea9aaa" event={"ID":"49b44719-fbc8-4fd5-9a70-b45ff547c813","Type":"ContainerStarted","Data":"1c183fe49df1eaac6d8efdc5deb54da489bb152460d03a4c6daa339d98704b78"} Oct 30 01:20:11.322811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:11.322775 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 01:20:11.323221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:11.322987 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" containerID="cri-o://99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8" gracePeriod=30 Oct 30 01:20:11.330915 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:11.330846 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8768-ea9aaa" podStartSLOduration=1.330832124 podStartE2EDuration="1.330832124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:11.329328068 +0000 UTC m=+2202.171888660" watchObservedRunningTime="2026-10-30 01:20:11.330832124 +0000 UTC m=+2202.173392716" Oct 30 01:20:12.162300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.162265 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8768-ea9aaa"] Oct 30 01:20:12.165581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.165432 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8768-ea9aaa"] Oct 30 01:20:12.314387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.314355 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8768-2e2b40"] Oct 30 01:20:12.315145 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.315126 5548 cpu_manager.go:414] "containerMap: removing container" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" containerName="test-curl-8768-ea9aaa" Oct 30 01:20:12.315218 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.315146 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" containerName="test-curl-8768-ea9aaa" Oct 30 01:20:12.315260 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.315236 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" containerName="test-curl-8768-ea9aaa" Oct 30 01:20:12.319054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.319037 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:12.332525 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.332497 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-2e2b40"] Oct 30 01:20:12.368569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.368532 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zqz\" (UniqueName: \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\") pod \"test-curl-8768-2e2b40\" (UID: \"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0\") " pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:12.469609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.469536 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zqz\" (UniqueName: \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\") pod \"test-curl-8768-2e2b40\" (UID: \"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0\") " pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:12.486673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.486635 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zqz\" (UniqueName: \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\") pod \"test-curl-8768-2e2b40\" (UID: \"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0\") " pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:12.631387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.631358 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:12.934626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:12.934584 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8768-2e2b40"] Oct 30 01:20:12.936575 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:20:12.936542 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35b06860_40c3_46c6_9bc2_1e0e4d6d82a0.slice/crio-eab5966063a0bcc29b8ac9baa9c9987265297d8af28dd92a60193eb21a5bd3b2 WatchSource:0}: Error finding container eab5966063a0bcc29b8ac9baa9c9987265297d8af28dd92a60193eb21a5bd3b2: Status 404 returned error can't find the container with id eab5966063a0bcc29b8ac9baa9c9987265297d8af28dd92a60193eb21a5bd3b2 Oct 30 01:20:13.304006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:13.303904 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-2e2b40" event={"ID":"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0","Type":"ContainerStarted","Data":"0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937"} Oct 30 01:20:13.304006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:13.303948 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8768-2e2b40" event={"ID":"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0","Type":"ContainerStarted","Data":"eab5966063a0bcc29b8ac9baa9c9987265297d8af28dd92a60193eb21a5bd3b2"} Oct 30 01:20:13.304245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:13.304058 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8768-ea9aaa" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" containerName="test-curl-8768-ea9aaa" containerID="cri-o://e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4" gracePeriod=2 Oct 30 01:20:13.335746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:13.335678 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8768-2e2b40" podStartSLOduration=1.3356634299999999 podStartE2EDuration="1.33566343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:13.334444511 +0000 UTC m=+2204.177005105" watchObservedRunningTime="2026-10-30 01:20:13.33566343 +0000 UTC m=+2204.178224022" Oct 30 01:20:14.170661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:14.170625 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8768-2e2b40"] Oct 30 01:20:14.174602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:14.174566 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8768-2e2b40"] Oct 30 01:20:15.317593 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:15.317561 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8768-2e2b40" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" containerName="test-curl-8768-2e2b40" containerID="cri-o://0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937" gracePeriod=2 Oct 30 01:20:15.500774 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:15.500747 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:15.613743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:15.613660 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\") pod \"49b44719-fbc8-4fd5-9a70-b45ff547c813\" (UID: \"49b44719-fbc8-4fd5-9a70-b45ff547c813\") " Oct 30 01:20:15.615884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:15.615832 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv" pod "49b44719-fbc8-4fd5-9a70-b45ff547c813" (UID: "49b44719-fbc8-4fd5-9a70-b45ff547c813"). InnerVolumeSpecName "kube-api-access-zhhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:15.715717 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:15.715683 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/49b44719-fbc8-4fd5-9a70-b45ff547c813-kube-api-access-zhhwv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.325478 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.325439 5548 generic.go:356] "Generic (PLEG): container finished" podID="49b44719-fbc8-4fd5-9a70-b45ff547c813" containerID="e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4" exitCode=137 Oct 30 01:20:16.326002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.325482 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:16.326002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.325511 5548 scope.go:122] "RemoveContainer" containerID="e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4" Oct 30 01:20:16.337266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.337246 5548 scope.go:122] "RemoveContainer" containerID="e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4" Oct 30 01:20:16.337562 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:16.337543 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4\": container with ID starting with e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4 not found: ID does not exist" containerID="e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4" Oct 30 01:20:16.337612 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.337573 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4"} err="failed to get container status \"e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4\": rpc error: code = NotFound desc = could not find container \"e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4\": container with ID starting with e07c1651644d0284d75c5d1cad59a6737f094408f1715a4573bbe0cd0f2074c4 not found: ID does not exist" Oct 30 01:20:16.347123 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:16.347087 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-ea9aaa\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:16.600613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.600588 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 01:20:16.675279 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:16.675232 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-ea9aaa\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" pod="opendatahub/test-curl-8768-ea9aaa" Oct 30 01:20:16.729558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729528 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.729747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729570 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.729747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729656 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.729747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729691 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.729747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729720 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729756 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729785 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729813 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729875 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\") pod \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\" (UID: \"4e733aae-b7be-48c5-afd1-4fdab74f24a8\") " Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729917 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:16.730038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.729965 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:16.730330 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.730162 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:16.730330 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.730252 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:16.730451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.730429 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:16.731306 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.731277 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.731434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.731312 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.731434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.731328 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.731434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.731342 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.731434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.731355 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.732468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.732446 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:16.732974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.732946 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:16.733076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.732970 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:16.733225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.733203 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm" pod "4e733aae-b7be-48c5-afd1-4fdab74f24a8" (UID: "4e733aae-b7be-48c5-afd1-4fdab74f24a8"). InnerVolumeSpecName "kube-api-access-kjllm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:16.832309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.832220 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.832309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.832256 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.832309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.832288 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e733aae-b7be-48c5-afd1-4fdab74f24a8-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:16.832309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:16.832297 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjllm\" (UniqueName: \"kubernetes.io/projected/4e733aae-b7be-48c5-afd1-4fdab74f24a8-kube-api-access-kjllm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:17.266241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.266207 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b44719-fbc8-4fd5-9a70-b45ff547c813" path="/var/lib/kubelet/pods/49b44719-fbc8-4fd5-9a70-b45ff547c813/volumes" Oct 30 01:20:17.335728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.335692 5548 generic.go:356] "Generic (PLEG): container finished" podID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerID="99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8" exitCode=0 Oct 30 01:20:17.336162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.335767 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" Oct 30 01:20:17.336162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.335779 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerDied","Data":"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8"} Oct 30 01:20:17.336162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.335836 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4" event={"ID":"4e733aae-b7be-48c5-afd1-4fdab74f24a8","Type":"ContainerDied","Data":"dbb14389094d54331cf7c3c0f9d57426326476883af45dd62c73eb64ffd947c9"} Oct 30 01:20:17.336162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.335877 5548 scope.go:122] "RemoveContainer" containerID="99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8" Oct 30 01:20:17.383503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.383453 5548 scope.go:122] "RemoveContainer" containerID="224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e" Oct 30 01:20:17.397746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.397716 5548 scope.go:122] "RemoveContainer" containerID="99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8" Oct 30 01:20:17.398106 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:17.398081 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8\": container with ID starting with 99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8 not found: ID does not exist" containerID="99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8" Oct 30 01:20:17.398194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.398120 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8"} err="failed to get container status \"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8\": rpc error: code = NotFound desc = could not find container \"99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8\": container with ID starting with 99f7d752c9b30f1cc755cc6a468925b7cc73b09f125ea271a08a2913018719a8 not found: ID does not exist" Oct 30 01:20:17.398194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.398145 5548 scope.go:122] "RemoveContainer" containerID="224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e" Oct 30 01:20:17.398431 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:17.398408 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e\": container with ID starting with 224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e not found: ID does not exist" containerID="224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e" Oct 30 01:20:17.398516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.398438 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e"} err="failed to get container status \"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e\": rpc error: code = NotFound desc = could not find container \"224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e\": container with ID starting with 224cf5c3862b991956480de5a93760b13540e912dca1225c6ffd849d0b847c2e not found: ID does not exist" Oct 30 01:20:17.412268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.412220 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 01:20:17.422056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.422027 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8ntx4"] Oct 30 01:20:17.651885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.646977 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:17.753896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.747624 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\" (UniqueName: \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\") pod \"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0\" (UID: \"35b06860-40c3-46c6-9bc2-1e0e4d6d82a0\") " Oct 30 01:20:17.753896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.753094 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz" pod "35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" (UID: "35b06860-40c3-46c6-9bc2-1e0e4d6d82a0"). InnerVolumeSpecName "kube-api-access-55zqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:17.853728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:17.853647 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zqz\" (UniqueName: \"kubernetes.io/projected/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0-kube-api-access-55zqz\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:20:18.361832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:18.360448 5548 generic.go:356] "Generic (PLEG): container finished" podID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" containerID="0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937" exitCode=137 Oct 30 01:20:18.361832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:18.360628 5548 scope.go:122] "RemoveContainer" containerID="0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937" Oct 30 01:20:18.361832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:18.360750 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:18.377627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:18.377602 5548 scope.go:122] "RemoveContainer" containerID="0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937" Oct 30 01:20:18.377967 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:18.377943 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937\": container with ID starting with 0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937 not found: ID does not exist" containerID="0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937" Oct 30 01:20:18.378043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:18.377981 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937"} err="failed to get container status \"0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937\": rpc error: code = NotFound desc = could not find container \"0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937\": container with ID starting with 0b92ee3fc772b6c8af5c1ed8d2c68e4c94103073ce894e804ae8917062731937 not found: ID does not exist" Oct 30 01:20:18.386415 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:18.386377 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-2e2b40\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:19.232204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.232162 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:20:19.232980 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.232960 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:19.232980 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.232980 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:19.233131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.233006 5548 cpu_manager.go:414] "containerMap: removing container" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" containerName="test-curl-8768-2e2b40" Oct 30 01:20:19.233131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.233015 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" containerName="test-curl-8768-2e2b40" Oct 30 01:20:19.233200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.233171 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:19.233200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.233183 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:19.233264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.233225 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" containerName="test-curl-8768-2e2b40" Oct 30 01:20:19.238310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.238290 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.252295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.252266 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-886adb-serving-cert\"" Oct 30 01:20:19.269595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.269563 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" path="/var/lib/kubelet/pods/35b06860-40c3-46c6-9bc2-1e0e4d6d82a0/volumes" Oct 30 01:20:19.270296 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.270273 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" path="/var/lib/kubelet/pods/4e733aae-b7be-48c5-afd1-4fdab74f24a8/volumes" Oct 30 01:20:19.271895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.270786 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:20:19.290402 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:19.290365 5548 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8768-2e2b40\" is forbidden: User \"system:node:ip-10-0-143-7.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-7.ec2.internal' and this object" podUID="35b06860-40c3-46c6-9bc2-1e0e4d6d82a0" pod="opendatahub/test-curl-8768-2e2b40" Oct 30 01:20:19.373309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.373277 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.373787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.373347 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2dd\" (UniqueName: \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.373787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.373545 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.373787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.373723 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.475465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.475423 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.475675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.475590 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.475675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.475654 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2dd\" (UniqueName: \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.475792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.475686 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.476469 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.476439 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.478198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.478177 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.478326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.478300 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.492688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.492621 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2dd\" (UniqueName: \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\") pod \"maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.553166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.553135 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:19.900544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.900515 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:20:19.902135 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:20:19.902107 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4c6579_8d82_4b02_9680_0a5e57686732.slice/crio-70c76c7f09f786ca381196ad112f5f3711f9c8decec8726828f18f57ab73855c WatchSource:0}: Error finding container 70c76c7f09f786ca381196ad112f5f3711f9c8decec8726828f18f57ab73855c: Status 404 returned error can't find the container with id 70c76c7f09f786ca381196ad112f5f3711f9c8decec8726828f18f57ab73855c Oct 30 01:20:19.903740 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:19.903719 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:20:20.407210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:20.406498 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" event={"ID":"eb4c6579-8d82-4b02-9680-0a5e57686732","Type":"ContainerStarted","Data":"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44"} Oct 30 01:20:20.407210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:20.406566 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" event={"ID":"eb4c6579-8d82-4b02-9680-0a5e57686732","Type":"ContainerStarted","Data":"70c76c7f09f786ca381196ad112f5f3711f9c8decec8726828f18f57ab73855c"} Oct 30 01:20:20.409013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:20.408933 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:20.446833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:20.446778 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" podStartSLOduration=1.123537853 podStartE2EDuration="1.446754287s" totalImagesPullingTime="323.216434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:20.44473101 +0000 UTC m=+2211.287291604" watchObservedRunningTime="2026-10-30 01:20:20.446754287 +0000 UTC m=+2211.289314880" Oct 30 01:20:25.435221 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:25.435181 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:20:27.427326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:27.427301 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:20:28.837798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.837765 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:20:28.838544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.838523 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:28.838643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.838546 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e733aae-b7be-48c5-afd1-4fdab74f24a8" containerName="istio-proxy" Oct 30 01:20:28.842269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.842249 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:28.866100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.866075 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-d81f0a-serving-cert\"" Oct 30 01:20:28.870038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.870007 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:20:28.902917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.902880 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:28.903087 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.903039 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92tb\" (UniqueName: \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:28.903144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.903128 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:28.903268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:28.903254 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.004973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.004931 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92tb\" (UniqueName: \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.005151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.005085 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.005214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.005154 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.005266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.005217 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.006026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.005940 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.007932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.007910 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.008086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.008070 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.022279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.022253 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92tb\" (UniqueName: \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\") pod \"maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.157167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.157135 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:29.468906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:29.468804 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:20:29.477780 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:20:29.477752 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24c0fee_e35a_4b0c_8b8f_538f15cbc68a.slice/crio-2029450e4bdfc4defcb675d251b41326439fe13904e01483d821d4a01749371a WatchSource:0}: Error finding container 2029450e4bdfc4defcb675d251b41326439fe13904e01483d821d4a01749371a: Status 404 returned error can't find the container with id 2029450e4bdfc4defcb675d251b41326439fe13904e01483d821d4a01749371a Oct 30 01:20:30.486778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:30.486743 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" event={"ID":"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a","Type":"ContainerStarted","Data":"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915"} Oct 30 01:20:30.487270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:30.486796 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" event={"ID":"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a","Type":"ContainerStarted","Data":"2029450e4bdfc4defcb675d251b41326439fe13904e01483d821d4a01749371a"} Oct 30 01:20:30.487270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:30.486981 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:30.534022 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:30.533956 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" podStartSLOduration=2.175391742 podStartE2EDuration="2.53393501s" totalImagesPullingTime="358.543268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:30.524980353 +0000 UTC m=+2221.367540959" watchObservedRunningTime="2026-10-30 01:20:30.53393501 +0000 UTC m=+2221.376495620" Oct 30 01:20:36.498546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:36.498512 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:20:55.428286 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:20:55.428239 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:20:56.867983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.865256 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:20:56.870533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.870506 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.885902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.884930 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-999ff9-openshift-default-dockercfg-8jtmv\"" Oct 30 01:20:56.899683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.899639 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:20:56.940344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940311 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940348 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940491 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940513 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940594 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940679 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940718 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940823 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:56.940910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:56.940854 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbd65\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.041848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.041808 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.041882 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbd65\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.041964 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.041992 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042096 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042126 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042164 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042236 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.042568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042266 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.043006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.042982 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.044003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.043536 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.044003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.043811 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.044003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.043836 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.044502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.044482 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.046166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.045970 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.046166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.046123 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.065728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.065698 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.066985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.066959 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbd65\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\") pod \"e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.195157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.195119 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:57.553943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.553682 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:20:57.556743 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:20:57.556709 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde533c29_45f0_49a8_9ee4_36637854c549.slice/crio-e1480cae4d34a1e66b1b6685b1de57e078e16b019de6584f63c5b7ee6b410aa0 WatchSource:0}: Error finding container e1480cae4d34a1e66b1b6685b1de57e078e16b019de6584f63c5b7ee6b410aa0: Status 404 returned error can't find the container with id e1480cae4d34a1e66b1b6685b1de57e078e16b019de6584f63c5b7ee6b410aa0 Oct 30 01:20:57.559328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.559294 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:20:57.559416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.559364 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:20:57.559416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.559394 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:20:57.672424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.672384 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerStarted","Data":"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0"} Oct 30 01:20:57.672602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:57.672459 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerStarted","Data":"e1480cae4d34a1e66b1b6685b1de57e078e16b019de6584f63c5b7ee6b410aa0"} Oct 30 01:20:58.196501 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:58.196463 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:58.203398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:58.203372 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:58.246940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:58.246872 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podStartSLOduration=2.2468424 podStartE2EDuration="2.2468424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:57.712939051 +0000 UTC m=+2248.555499672" watchObservedRunningTime="2026-10-30 01:20:58.2468424 +0000 UTC m=+2249.089402993" Oct 30 01:20:58.681797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:58.681768 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:20:58.683076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:20:58.683051 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:21:02.079789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.079750 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:21:02.085386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.085358 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.103056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.103017 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-886adb-openshift-default-dockercfg-l6rhb\"" Oct 30 01:21:02.117691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.117230 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:21:02.214259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214218 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214264 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214362 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69w2j\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214408 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214630 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214739 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214809 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214849 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.214909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.214897 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.315824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.315798 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.315830 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.315870 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.315984 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.316020 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.316053 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69w2j\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316183 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.316087 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.316326 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.316548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.316515 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.317093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.317065 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.317197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.317107 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.317197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.317143 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.317272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.317190 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.317417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.317400 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.318436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.318415 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.319144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.319122 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.337136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.337074 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.337136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.337104 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69w2j\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.410937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.410906 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:02.728326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.728301 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:21:02.729381 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:02.729353 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711541f2_3cef_4eda_8ea8_d0f3fa0daeda.slice/crio-48f1367b57e90a9fca5f27ac7833e5256e16da70abd86179333f1da15def1670 WatchSource:0}: Error finding container 48f1367b57e90a9fca5f27ac7833e5256e16da70abd86179333f1da15def1670: Status 404 returned error can't find the container with id 48f1367b57e90a9fca5f27ac7833e5256e16da70abd86179333f1da15def1670 Oct 30 01:21:02.731980 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.731947 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:02.732049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.732013 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:02.732049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:02.732040 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:03.732474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:03.732440 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" event={"ID":"711541f2-3cef-4eda-8ea8-d0f3fa0daeda","Type":"ContainerStarted","Data":"c32e620c7e909661c21c4ab721c5ff237cb42d6213a878dad470ba8065a68488"} Oct 30 01:21:03.733008 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:03.732495 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" event={"ID":"711541f2-3cef-4eda-8ea8-d0f3fa0daeda","Type":"ContainerStarted","Data":"48f1367b57e90a9fca5f27ac7833e5256e16da70abd86179333f1da15def1670"} Oct 30 01:21:03.788884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:03.785095 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podStartSLOduration=1.785074831 podStartE2EDuration="1.785074831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:03.784923007 +0000 UTC m=+2254.627483622" watchObservedRunningTime="2026-10-30 01:21:03.785074831 +0000 UTC m=+2254.627635425" Oct 30 01:21:04.411826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:04.411785 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:04.413373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:04.413350 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:21:04.413438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:04.413418 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:21:05.416226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:05.416192 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:21:05.416785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:05.416711 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:21:06.412314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:06.412278 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:21:06.412524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:06.412349 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:21:07.412163 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:07.412128 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:21:07.412727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:07.412188 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:21:08.412633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.412591 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:21:08.413092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.412663 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:21:08.451848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.451818 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:21:08.457606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.456621 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.492269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.492236 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2be695-openshift-default-dockercfg-f5bng\"" Oct 30 01:21:08.529002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.528943 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:21:08.656286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.656241 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.656983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.656957 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.657979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.657957 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz8j\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.658262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.658241 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.658432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.658417 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.658600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.658582 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.658952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.658932 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.659139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.659124 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.659300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.659285 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760346 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760393 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760413 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760439 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760475 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760500 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz8j\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.760536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760531 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.761102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760549 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.761102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.760574 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.761285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.761260 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.762065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.762039 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.763329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.763307 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.763883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.763834 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.764220 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.764194 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.766298 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.766277 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.801882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.786651 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.814910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.807462 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:08.818897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:08.817679 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz8j\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:09.110817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.110732 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:09.417506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.417473 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:09.417995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.417963 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:09.418884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.418848 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:21:09.446437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.446403 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:21:09.448219 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:09.448184 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04658015_a02c_4f70_a006_71a6e1f10a5f.slice/crio-78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b WatchSource:0}: Error finding container 78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b: Status 404 returned error can't find the container with id 78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b Oct 30 01:21:09.451215 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.451181 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:09.451301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.451252 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:09.451301 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.451281 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:09.798974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.798899 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" event={"ID":"04658015-a02c-4f70-a006-71a6e1f10a5f","Type":"ContainerStarted","Data":"6411f28a7634d3124d0bf29331ad2c7802acadb07b083bbc43e529b4f8b7acd5"} Oct 30 01:21:09.798974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.798967 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" event={"ID":"04658015-a02c-4f70-a006-71a6e1f10a5f","Type":"ContainerStarted","Data":"78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b"} Oct 30 01:21:09.842577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:09.842523 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podStartSLOduration=1.842508401 podStartE2EDuration="1.842508401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:09.835791709 +0000 UTC m=+2260.678352305" watchObservedRunningTime="2026-10-30 01:21:09.842508401 +0000 UTC m=+2260.685068988" Oct 30 01:21:10.111120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:10.111013 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:10.112425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:10.112400 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:10.112495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:10.112448 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:11.111491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:11.111454 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:11.112003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:11.111534 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:12.111478 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:12.111443 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:12.111663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:12.111495 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:13.111280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:13.111248 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:13.111491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:13.111308 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:14.111972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:14.111881 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:14.111972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:14.111937 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:15.112179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.111121 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:15.112179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.111185 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:15.914068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.914034 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:21:15.918784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.918755 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:15.943842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.943814 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d32079-serving-cert\"" Oct 30 01:21:15.947262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.947229 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:21:15.968270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.968224 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:15.968430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.968400 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:15.968560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.968541 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:15.968629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:15.968618 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9lb\" (UniqueName: \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.069942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.069909 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9lb\" (UniqueName: \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.070134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.069997 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.070208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.070130 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.070208 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:21:16.070153 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-serving-cert: secret "maas-api-e2e-ipp-a-d32079-serving-cert" not found Oct 30 01:21:16.070297 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:21:16.070263 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls podName:90826748-a433-453c-9bfb-a0a26ac1b4b9 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:16.570242061 +0000 UTC m=+2267.412802640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls") pod "maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" (UID: "90826748-a433-453c-9bfb-a0a26ac1b4b9") : secret "maas-api-e2e-ipp-a-d32079-serving-cert" not found Oct 30 01:21:16.070297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.070283 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.070970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.070943 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.073053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.073028 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.102903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.102829 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9lb\" (UniqueName: \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.112127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.112096 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:21:16.112303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.112153 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:21:16.578839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.578802 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.581624 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.581593 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:16.834546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:16.834461 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:17.117440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.117373 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:17.117825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.117656 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:17.118553 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.118527 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:21:17.140636 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:17.140608 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90826748_a433_453c_9bfb_a0a26ac1b4b9.slice/crio-7388744172c6aff135f36f2adddeba515c0c77ac3828b5ba8f40fc8724fb0ffb WatchSource:0}: Error finding container 7388744172c6aff135f36f2adddeba515c0c77ac3828b5ba8f40fc8724fb0ffb: Status 404 returned error can't find the container with id 7388744172c6aff135f36f2adddeba515c0c77ac3828b5ba8f40fc8724fb0ffb Oct 30 01:21:17.143450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.143429 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:21:17.871090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.870981 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" event={"ID":"90826748-a433-453c-9bfb-a0a26ac1b4b9","Type":"ContainerStarted","Data":"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb"} Oct 30 01:21:17.871270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.871145 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:17.871270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.871167 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" event={"ID":"90826748-a433-453c-9bfb-a0a26ac1b4b9","Type":"ContainerStarted","Data":"7388744172c6aff135f36f2adddeba515c0c77ac3828b5ba8f40fc8724fb0ffb"} Oct 30 01:21:17.911139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:17.911039 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" podStartSLOduration=2.544021296 podStartE2EDuration="2.911022364s" totalImagesPullingTime="367.001068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:17.907851453 +0000 UTC m=+2268.750412049" watchObservedRunningTime="2026-10-30 01:21:17.911022364 +0000 UTC m=+2268.753582950" Oct 30 01:21:24.892918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:24.892852 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:21:25.428779 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:21:25.428744 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:21:47.192639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.192601 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:21:47.198698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.198590 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.224503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.224469 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:21:47.235330 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.235202 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:21:47.235530 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.235339 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.255468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.255434 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:21:47.361511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.361466 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.361680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.361592 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.361752 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.361714 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.361960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.361815 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.362060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362008 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.362060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362038 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.362148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362118 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7lt\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.362184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362146 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.362598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362515 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.362598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362596 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.362780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362681 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.362780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362770 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.362900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.362797 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvpx\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.363139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.363114 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.363272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.363164 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.363272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.363190 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.363272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.363222 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.363445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.363343 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.464696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464605 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.464696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464643 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvpx\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464779 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464822 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464847 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464921 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.464967 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465025 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465075 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465157 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465184 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465228 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465250 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465317 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7lt\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465343 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465374 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465420 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465425 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465510 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.465632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465566 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.466028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465695 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.466028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465695 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.466028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465845 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.466028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.465952 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.466028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.466018 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.466308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.466273 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.466361 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.466241 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.466524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.466500 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.467643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.467620 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.467822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.467806 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.468229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.468207 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.468500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.468483 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.482109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.482081 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.482515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.482494 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7lt\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.482661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.482640 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvpx\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.482842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.482825 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.514767 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.514736 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:47.564921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.564848 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:47.833367 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.833334 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:21:47.834104 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:47.834074 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bf83eb_3cc0_49ac_a48b_453f2cf52673.slice/crio-945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6 WatchSource:0}: Error finding container 945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6: Status 404 returned error can't find the container with id 945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6 Oct 30 01:21:47.837226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.837193 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:47.837308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.837260 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:47.837308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.837294 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:47.892057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.892034 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:21:47.892641 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:47.892612 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991987ed_57be_4005_bb55_691e1c9fa3c9.slice/crio-1b69a5d66f10a6b00dc95e106921f3835d835bd0bbfa248086000c8b864b1dec WatchSource:0}: Error finding container 1b69a5d66f10a6b00dc95e106921f3835d835bd0bbfa248086000c8b864b1dec: Status 404 returned error can't find the container with id 1b69a5d66f10a6b00dc95e106921f3835d835bd0bbfa248086000c8b864b1dec Oct 30 01:21:47.895782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.895751 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:47.895887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.895812 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:47.895887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:47.895841 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:48.135009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.134887 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" event={"ID":"991987ed-57be-4005-bb55-691e1c9fa3c9","Type":"ContainerStarted","Data":"7f3d32dea1bf61fc939e8329ee6005cfc8e8ebed12b55c99c8d1cd58e6198e87"} Oct 30 01:21:48.135009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.134958 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" event={"ID":"991987ed-57be-4005-bb55-691e1c9fa3c9","Type":"ContainerStarted","Data":"1b69a5d66f10a6b00dc95e106921f3835d835bd0bbfa248086000c8b864b1dec"} Oct 30 01:21:48.137729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.137702 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" event={"ID":"81bf83eb-3cc0-49ac-a48b-453f2cf52673","Type":"ContainerStarted","Data":"bdcf5c171c26c70e1a0e230e0e96ecb2e3baf299983c6ad883eb8ea16177d655"} Oct 30 01:21:48.137885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.137763 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" event={"ID":"81bf83eb-3cc0-49ac-a48b-453f2cf52673","Type":"ContainerStarted","Data":"945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6"} Oct 30 01:21:48.174560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.174498 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podStartSLOduration=1.174477901 podStartE2EDuration="1.174477901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:48.171665849 +0000 UTC m=+2299.014226443" watchObservedRunningTime="2026-10-30 01:21:48.174477901 +0000 UTC m=+2299.017038496" Oct 30 01:21:48.215212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.215152 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podStartSLOduration=1.215133038 podStartE2EDuration="1.215133038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:48.213407359 +0000 UTC m=+2299.055967953" watchObservedRunningTime="2026-10-30 01:21:48.215133038 +0000 UTC m=+2299.057693632" Oct 30 01:21:48.515133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.515097 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:48.516944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.516917 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:48.517061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.516967 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:48.565932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.565896 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:48.567194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.567172 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:48.567316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:48.567218 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:49.516884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:49.516535 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:49.516884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:49.516597 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:49.565358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:49.565329 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:49.565561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:49.565541 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:50.518037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:50.518002 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:50.518648 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:50.518579 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:50.565631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:50.565586 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:50.565783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:50.565646 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:51.519372 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:51.519327 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:51.519959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:51.519405 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:51.565484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:51.565455 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:51.565643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:51.565513 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:52.518286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:52.518253 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:52.518471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:52.518437 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:52.565548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:52.565504 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:52.566036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:52.565576 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:53.524891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.519284 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:53.524891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.519363 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:53.579889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.572012 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:53.579889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.572088 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:53.584209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.584175 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:21:53.618393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.618364 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.618722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.618542 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751318 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751387 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751453 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751512 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751580 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72ln\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751615 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751725 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751810 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.754911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.751850 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.858019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.857685 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.858320 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.858296 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72ln\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.858445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.858431 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.858727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.858620 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.858919 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859001 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859123 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859338 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859364 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859465 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859510 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859640 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859583 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.859985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.859947 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.860076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.860010 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.862454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.862241 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.862910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.862850 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.955438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.955007 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72ln\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.957892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.956617 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:53.962425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:53.961140 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:54.421990 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:21:54.421937 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3740ffd4_a844_4c13_8914_ec81a7b5c129.slice/crio-9dee578058aa1f73855520763bd5da84757430e181947c12a95d92a4c52d5571 WatchSource:0}: Error finding container 9dee578058aa1f73855520763bd5da84757430e181947c12a95d92a4c52d5571: Status 404 returned error can't find the container with id 9dee578058aa1f73855520763bd5da84757430e181947c12a95d92a4c52d5571 Oct 30 01:21:54.422729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.422694 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:21:54.425503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.425470 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:54.425567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.425535 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:54.425617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.425565 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:21:54.521886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.515747 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:21:54.521886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.515807 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 01:21:54.565597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.565464 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:21:54.565597 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:54.565525 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:21:55.250928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.250891 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" event={"ID":"3740ffd4-a844-4c13-8914-ec81a7b5c129","Type":"ContainerStarted","Data":"bb82907693e1e4bf08824c6c28b97e05bf302393f0b61e8e89147e1a8fcb9b55"} Oct 30 01:21:55.251417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.250948 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" event={"ID":"3740ffd4-a844-4c13-8914-ec81a7b5c129","Type":"ContainerStarted","Data":"9dee578058aa1f73855520763bd5da84757430e181947c12a95d92a4c52d5571"} Oct 30 01:21:55.287716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.287666 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podStartSLOduration=2.287651683 podStartE2EDuration="2.287651683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:55.286957507 +0000 UTC m=+2306.129518117" watchObservedRunningTime="2026-10-30 01:21:55.287651683 +0000 UTC m=+2306.130212275" Oct 30 01:21:55.429981 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:21:55.429941 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:21:55.532954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.525543 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:55.534118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.534084 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:55.537060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.537034 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:21:55.578685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.578644 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:55.581083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.581058 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:55.582189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.582167 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:21:55.963423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.963392 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:21:55.965141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.965097 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:21:55.965292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:55.965177 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:21:56.963568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:56.963534 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:21:56.964065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:56.963609 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:21:57.963804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:57.963771 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:21:57.964204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:57.963836 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:21:58.963059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:58.963024 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:21:58.963247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:58.963091 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:21:59.962981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:59.962939 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:21:59.963429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:21:59.963009 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:22:00.963882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:00.963827 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:22:00.964323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:00.963936 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:22:01.969326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:01.969290 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:22:01.969819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:01.969731 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:22:01.972110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:01.972088 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:22:02.604408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.604375 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:22:02.609043 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.609021 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.625585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.625555 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4be164-serving-cert\"" Oct 30 01:22:02.626946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.626899 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:22:02.689741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.689706 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.689967 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.689751 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.690033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.690004 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.690140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.690120 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.790987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.790953 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.791157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.791040 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.791157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.791105 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.791157 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.791124 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.791837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.791812 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.793711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.793682 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.793822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.793810 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.810644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.810621 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\") pod \"maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:02.924869 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:02.924805 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:03.269413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:03.269389 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:22:03.269903 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:22:03.269881 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc37bf4c_08b6_4013_a029_58429ad97f72.slice/crio-607b574376c0d6285a2562ea50b4f995dbc035ae70f91060396d003e4a071f82 WatchSource:0}: Error finding container 607b574376c0d6285a2562ea50b4f995dbc035ae70f91060396d003e4a071f82: Status 404 returned error can't find the container with id 607b574376c0d6285a2562ea50b4f995dbc035ae70f91060396d003e4a071f82 Oct 30 01:22:03.344968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:03.344932 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" event={"ID":"bc37bf4c-08b6-4013-a029-58429ad97f72","Type":"ContainerStarted","Data":"607b574376c0d6285a2562ea50b4f995dbc035ae70f91060396d003e4a071f82"} Oct 30 01:22:04.353957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:04.353890 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" event={"ID":"bc37bf4c-08b6-4013-a029-58429ad97f72","Type":"ContainerStarted","Data":"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0"} Oct 30 01:22:04.354379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:04.353979 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:04.392276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:04.392220 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" podStartSLOduration=2.060024136 podStartE2EDuration="2.392206909s" totalImagesPullingTime="332.182773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:04.387921794 +0000 UTC m=+2315.230482404" watchObservedRunningTime="2026-10-30 01:22:04.392206909 +0000 UTC m=+2315.234767499" Oct 30 01:22:10.367661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:10.367629 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:22:25.436577 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:25.436537 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:22:34.642750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.642703 5548 generic.go:356] "Generic (PLEG): container finished" podID="de533c29-45f0-49a8-9ee4-36637854c549" containerID="62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0" exitCode=0 Oct 30 01:22:34.643916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.642966 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerDied","Data":"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0"} Oct 30 01:22:34.643916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.643548 5548 scope.go:122] "RemoveContainer" containerID="62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0" Oct 30 01:22:34.646949 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.646908 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:34.647056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.647006 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:34.647099 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:34.647052 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:35.019613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.019511 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:22:35.024671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.024645 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.054095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.054064 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d32079-dockercfg-7z8kk\"" Oct 30 01:22:35.054472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.054448 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d32079\"" Oct 30 01:22:35.057395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.057365 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:22:35.133054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.133015 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.133261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.133241 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958sj\" (UniqueName: \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.234536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.234498 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958sj\" (UniqueName: \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.234750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.234674 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.235494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.235466 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.252306 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.252276 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958sj\" (UniqueName: \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\") pod \"payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.342903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.342797 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:35.652997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.652966 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:22:35.653454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:35.653425 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerStarted","Data":"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782"} Oct 30 01:22:35.654253 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:22:35.654219 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2376c94c_298d_477c_85b3_9201f67def0b.slice/crio-9cf1a737cb692ee1684d88f1d32d846ff29285bb39a9590c47d7a00c1bb5319f WatchSource:0}: Error finding container 9cf1a737cb692ee1684d88f1d32d846ff29285bb39a9590c47d7a00c1bb5319f: Status 404 returned error can't find the container with id 9cf1a737cb692ee1684d88f1d32d846ff29285bb39a9590c47d7a00c1bb5319f Oct 30 01:22:36.195775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:36.195735 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:22:36.196036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:36.196005 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:22:36.196116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:36.196051 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:22:36.666631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:36.666599 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" event={"ID":"2376c94c-298d-477c-85b3-9201f67def0b","Type":"ContainerStarted","Data":"9cf1a737cb692ee1684d88f1d32d846ff29285bb39a9590c47d7a00c1bb5319f"} Oct 30 01:22:37.205884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:37.205249 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:22:37.205884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:37.205316 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:22:37.705901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:37.702729 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" event={"ID":"2376c94c-298d-477c-85b3-9201f67def0b","Type":"ContainerStarted","Data":"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7"} Oct 30 01:22:37.705901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:37.702811 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:37.739814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:37.739754 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" podStartSLOduration=1.8594942890000001 podStartE2EDuration="3.73974082s" totalImagesPullingTime="1.880246531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:37.738289885 +0000 UTC m=+2348.580850478" watchObservedRunningTime="2026-10-30 01:22:37.73974082 +0000 UTC m=+2348.582301423" Oct 30 01:22:38.196410 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:38.196370 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:22:38.196625 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:38.196447 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:22:39.195885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:39.195656 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:22:39.195885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:39.195730 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:22:40.196445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:40.196357 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:22:40.196445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:40.196421 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:22:41.248766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:41.248725 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:22:41.249377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:41.249199 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:22:41.250675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:41.250652 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:22:45.628085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.628053 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:22:45.632002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.631978 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.647032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.647011 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b2a05c-serving-cert\"" Oct 30 01:22:45.661516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.661483 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:22:45.727747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.727716 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9qw\" (UniqueName: \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.727937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.727757 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.727937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.727898 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.727937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.727921 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.829261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.829217 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9qw\" (UniqueName: \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.829449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.829289 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.829449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.829400 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.829449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.829439 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.829793 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:45.829766 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-serving-cert: secret "maas-api-e2e-ipp-b-b2a05c-serving-cert" not found Oct 30 01:22:45.830898 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:45.829847 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls podName:b91fc1d5-3914-4e9f-9396-fa54b88f0065 nodeName:}" failed. No retries permitted until 2026-10-30 01:22:46.329824181 +0000 UTC m=+2357.172384771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls") pod "maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065") : secret "maas-api-e2e-ipp-b-b2a05c-serving-cert" not found Oct 30 01:22:45.831006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.830933 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.832772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.832744 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:45.853696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:45.853663 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9qw\" (UniqueName: \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:46.336221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:46.336179 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:46.336438 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:46.336326 5548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-serving-cert: secret "maas-api-e2e-ipp-b-b2a05c-serving-cert" not found Oct 30 01:22:46.336438 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:46.336393 5548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls podName:b91fc1d5-3914-4e9f-9396-fa54b88f0065 nodeName:}" failed. No retries permitted until 2026-10-30 01:22:47.336374865 +0000 UTC m=+2358.178935440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls") pod "maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065") : secret "maas-api-e2e-ipp-b-b2a05c-serving-cert" not found Oct 30 01:22:47.351360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:47.351316 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:47.353920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:47.353896 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:47.444978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:47.444948 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:47.759244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:47.759211 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:22:47.759443 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:22:47.759416 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91fc1d5_3914_4e9f_9396_fa54b88f0065.slice/crio-bdfda1cc89abc73624caf827119a3e9164256b0767404c25fefb3c583494e178 WatchSource:0}: Error finding container bdfda1cc89abc73624caf827119a3e9164256b0767404c25fefb3c583494e178: Status 404 returned error can't find the container with id bdfda1cc89abc73624caf827119a3e9164256b0767404c25fefb3c583494e178 Oct 30 01:22:47.802652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:47.802590 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" event={"ID":"b91fc1d5-3914-4e9f-9396-fa54b88f0065","Type":"ContainerStarted","Data":"bdfda1cc89abc73624caf827119a3e9164256b0767404c25fefb3c583494e178"} Oct 30 01:22:48.715468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:48.715437 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:22:48.816657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:48.816616 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" event={"ID":"b91fc1d5-3914-4e9f-9396-fa54b88f0065","Type":"ContainerStarted","Data":"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43"} Oct 30 01:22:48.816954 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:48.816935 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:48.861893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:48.861128 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" podStartSLOduration=3.507683891 podStartE2EDuration="3.861107498s" totalImagesPullingTime="353.423607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:48.856032547 +0000 UTC m=+2359.698593139" watchObservedRunningTime="2026-10-30 01:22:48.861107498 +0000 UTC m=+2359.703668092" Oct 30 01:22:50.690283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.690249 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:22:50.695584 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.695558 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.719375 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.719346 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e92648-openshift-default-dockercfg-zt4vb\"" Oct 30 01:22:50.742454 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.742409 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:22:50.811790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.811757 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.811876 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.811982 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812056 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812092 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812140 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812332 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812253 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812332 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812310 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.812495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.812456 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrmr\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913596 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913655 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913708 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrmr\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913822 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913943 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.913977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.913969 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914012 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914047 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914095 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914125 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914333 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914312 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914421 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.914500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.914479 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.915047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.915022 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.916304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.916282 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.916575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.916556 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.933953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.933930 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:50.934159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:50.934139 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrmr\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:51.014922 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.014817 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:51.349244 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:22:51.349213 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2847509_9131_43e1_9957_eb18b46773bb.slice/crio-5b0d3e7d37a7a9e583485b8e1773fa379736c6a49b21bea5fbdb0054cfce9430 WatchSource:0}: Error finding container 5b0d3e7d37a7a9e583485b8e1773fa379736c6a49b21bea5fbdb0054cfce9430: Status 404 returned error can't find the container with id 5b0d3e7d37a7a9e583485b8e1773fa379736c6a49b21bea5fbdb0054cfce9430 Oct 30 01:22:51.349784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.349759 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:22:51.352348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.352316 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:51.352442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.352380 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:51.352442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.352410 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:51.840116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.840080 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" event={"ID":"d2847509-9131-43e1-9957-eb18b46773bb","Type":"ContainerStarted","Data":"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0"} Oct 30 01:22:51.840566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.840145 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" event={"ID":"d2847509-9131-43e1-9957-eb18b46773bb","Type":"ContainerStarted","Data":"5b0d3e7d37a7a9e583485b8e1773fa379736c6a49b21bea5fbdb0054cfce9430"} Oct 30 01:22:51.881655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:51.881600 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podStartSLOduration=1.881580056 podStartE2EDuration="1.881580056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:51.878576041 +0000 UTC m=+2362.721136644" watchObservedRunningTime="2026-10-30 01:22:51.881580056 +0000 UTC m=+2362.724140650" Oct 30 01:22:52.015585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:52.015550 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:52.017418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:52.017388 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:52.017534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:52.017442 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:53.015941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.015907 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:53.016426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.015965 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:53.759953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.759921 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:22:53.765081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.765059 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.803750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.803719 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:22:53.855618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.855584 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.855814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.855675 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.855814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.855720 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.855986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.855911 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.856049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.856008 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.856049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.856037 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.856156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.856056 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.856227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.856208 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.856315 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.856261 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5nk\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.957934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.957898 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.957985 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5nk\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958079 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958103 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958135 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958282 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958388 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958436 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958464 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958473 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958663 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.958977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.958942 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.959127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.959090 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.959351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.959333 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.960587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.960567 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.960839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.960824 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.979885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.976427 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:53.979885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:53.978779 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5nk\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:54.016514 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.016437 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:54.016514 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.016494 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:54.110057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.110024 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:54.466727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.466685 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:22:54.508339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.508296 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:54.508464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.508393 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:54.508464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.508453 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:54.974891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:54.954766 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:22:55.071898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:55.070040 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:55.071898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:55.070097 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:55.358883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:55.358770 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" event={"ID":"0e41f202-179e-47fc-b776-4c744541606f","Type":"ContainerStarted","Data":"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc"} Oct 30 01:22:55.359241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:55.359215 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" event={"ID":"0e41f202-179e-47fc-b776-4c744541606f","Type":"ContainerStarted","Data":"d2ea1850d702d1f04d1587ee4e7601ca295cfcda2d5d7d4c1be0615255c94755"} Oct 30 01:22:55.389909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:55.389826 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podStartSLOduration=2.389805163 podStartE2EDuration="2.389805163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:55.354252284 +0000 UTC m=+2366.196812901" watchObservedRunningTime="2026-10-30 01:22:55.389805163 +0000 UTC m=+2366.232365756" Oct 30 01:22:55.473522 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:22:55.468313 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:22:56.018923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.018268 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:56.018923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.018329 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:56.119487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.112987 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:22:56.119487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.113104 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:22:56.119487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.113145 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:22:56.919889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.918923 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:22:56.926538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.926508 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:56.929297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.929202 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:22:56.941898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:56.940325 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-262230-openshift-default-dockercfg-ks9dj\"" Oct 30 01:22:57.024889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.024061 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:57.024889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.024125 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032056 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032125 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032258 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032294 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhc8j\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032359 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032399 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032464 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032503 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.045880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.032535 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.114889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.112279 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:22:57.114889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.112337 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134209 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134299 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134338 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134370 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134447 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134499 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134639 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134666 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhc8j\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.134735 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.135919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.135271 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.136850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.135963 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.136936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.136850 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.137887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.137111 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.137887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.137355 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.141583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.141503 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.141735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.141614 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.153829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.153801 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.154684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.154667 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhc8j\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.244908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.244799 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:57.646726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.646694 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:22:57.662846 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:22:57.662480 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df2711e_8717_4dd7_85a6_b750c51ba747.slice/crio-4d511a8852b43c2d2ec6a4bc57ac0ab4fa28b3d6f6442103e14b2270c4ebe765 WatchSource:0}: Error finding container 4d511a8852b43c2d2ec6a4bc57ac0ab4fa28b3d6f6442103e14b2270c4ebe765: Status 404 returned error can't find the container with id 4d511a8852b43c2d2ec6a4bc57ac0ab4fa28b3d6f6442103e14b2270c4ebe765 Oct 30 01:22:57.666687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.666610 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:57.666797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.666711 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:57.666797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:57.666754 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:22:58.016996 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.016630 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 01:22:58.016996 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.016694 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 01:22:58.110563 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.110519 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:22:58.110766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.110587 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:22:58.343486 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.343392 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" event={"ID":"8df2711e-8717-4dd7-85a6-b750c51ba747","Type":"ContainerStarted","Data":"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43"} Oct 30 01:22:58.344123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.344086 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" event={"ID":"8df2711e-8717-4dd7-85a6-b750c51ba747","Type":"ContainerStarted","Data":"4d511a8852b43c2d2ec6a4bc57ac0ab4fa28b3d6f6442103e14b2270c4ebe765"} Oct 30 01:22:58.400994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:58.400931 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podStartSLOduration=2.40091194 podStartE2EDuration="2.40091194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:58.396081349 +0000 UTC m=+2369.238641964" watchObservedRunningTime="2026-10-30 01:22:58.40091194 +0000 UTC m=+2369.243472534" Oct 30 01:22:59.027153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.027122 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:59.027610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.027573 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:59.028300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.028281 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:22:59.119267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.119234 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:22:59.119541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.119512 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:22:59.246060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.246028 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:22:59.247432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.247398 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:22:59.247568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:22:59.247459 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:00.111257 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:00.111222 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:23:00.111839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:00.111807 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:23:00.245553 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:00.245516 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:23:00.245750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:00.245571 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:01.115886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:01.114957 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:23:01.115886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:01.115023 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 01:23:01.245905 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:01.245820 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:23:01.246127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:01.245915 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:02.139224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:02.139187 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:23:02.141902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:02.140475 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:23:02.141902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:02.141295 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:23:02.245523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:02.245490 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:23:02.245694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:02.245547 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:03.246102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:03.246057 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:23:03.246595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:03.246122 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:04.245409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:04.245376 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:23:04.245575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:04.245431 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 01:23:05.250090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:05.250062 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:23:05.250531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:05.250513 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:23:05.251347 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:05.251327 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:23:25.528824 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:23:25.528777 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:23:29.820924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:29.816802 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:23:30.033921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.006426 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:23:30.033921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.006558 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.046895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.041118 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be164-dockercfg-bdrv9\"" Oct 30 01:23:30.070606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.070571 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be164\"" Oct 30 01:23:30.144060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.144019 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9gx\" (UniqueName: \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.144291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.144172 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.190464 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.190432 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:23:30.198246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.198220 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:23:30.250481 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.249908 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9gx\" (UniqueName: \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.252494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.252451 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.254434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.254412 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.295920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.290596 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9gx\" (UniqueName: \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\") pod \"payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.425117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.425083 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:30.735965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:30.735938 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:23:30.736903 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:23:30.736853 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdec348_3d68_454d_9c07_18a6e0d90a49.slice/crio-bff35155675c1adb4aa525956367a38e71317229a2feaf0a75c092950766b32f WatchSource:0}: Error finding container bff35155675c1adb4aa525956367a38e71317229a2feaf0a75c092950766b32f: Status 404 returned error can't find the container with id bff35155675c1adb4aa525956367a38e71317229a2feaf0a75c092950766b32f Oct 30 01:23:31.124415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:31.124314 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" event={"ID":"3bdec348-3d68-454d-9c07-18a6e0d90a49","Type":"ContainerStarted","Data":"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153"} Oct 30 01:23:31.124415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:31.124379 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" event={"ID":"3bdec348-3d68-454d-9c07-18a6e0d90a49","Type":"ContainerStarted","Data":"bff35155675c1adb4aa525956367a38e71317229a2feaf0a75c092950766b32f"} Oct 30 01:23:31.124962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:31.124436 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:31.158124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:31.158060 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" podStartSLOduration=2.158042066 podStartE2EDuration="2.158042066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:31.156885448 +0000 UTC m=+2401.999446043" watchObservedRunningTime="2026-10-30 01:23:31.158042066 +0000 UTC m=+2402.000602660" Oct 30 01:23:35.829880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.829825 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:23:35.834643 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.834623 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.864536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.864503 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:23:35.886410 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.886374 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.886929 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.886430 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.886929 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.886500 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.886929 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.886558 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.887131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.886842 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvkv\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.887131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.887118 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.887252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.887157 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.887252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.887223 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.887378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.887365 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.988821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.988786 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.988837 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.988886 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.988946 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.988997 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989185 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvkv\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989270 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989316 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989346 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989596 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.989835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.989809 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.990112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.990085 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.990112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.990118 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.990496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.990476 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.991696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.991674 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:35.992067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:35.992052 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:36.006201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.006183 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:36.006505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.006487 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvkv\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:36.161220 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.161189 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:36.479421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.479298 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:23:36.482581 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:23:36.482556 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc28590_5ca5_43fa_b2c9_29b67c89155d.slice/crio-3979467eeb262289aa01e512d1365a44cbc269222631ed83cf6d154ed585de8a WatchSource:0}: Error finding container 3979467eeb262289aa01e512d1365a44cbc269222631ed83cf6d154ed585de8a: Status 404 returned error can't find the container with id 3979467eeb262289aa01e512d1365a44cbc269222631ed83cf6d154ed585de8a Oct 30 01:23:36.485606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.485572 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:36.485688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.485641 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:36.485688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:36.485683 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:37.210661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:37.210617 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" event={"ID":"abc28590-5ca5-43fa-b2c9-29b67c89155d","Type":"ContainerStarted","Data":"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900"} Oct 30 01:23:37.211167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:37.210732 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" event={"ID":"abc28590-5ca5-43fa-b2c9-29b67c89155d","Type":"ContainerStarted","Data":"3979467eeb262289aa01e512d1365a44cbc269222631ed83cf6d154ed585de8a"} Oct 30 01:23:38.162370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:38.162328 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:38.163606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:38.163577 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:23:38.163756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:38.163640 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:23:39.163240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:39.163209 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:23:39.163743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:39.163277 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:23:40.164973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:40.164935 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:23:40.165670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:40.165606 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:23:41.162168 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:41.162132 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:23:41.162353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:41.162191 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:23:42.135100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:42.135069 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:23:42.161817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:42.161790 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:23:42.161999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:42.161849 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:23:42.194885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:42.194789 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podStartSLOduration=7.194769353 podStartE2EDuration="7.194769353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:37.270131896 +0000 UTC m=+2408.112692490" watchObservedRunningTime="2026-10-30 01:23:42.194769353 +0000 UTC m=+2413.037329946" Oct 30 01:23:43.167001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:43.166971 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:43.167411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:43.167393 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:43.168105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:43.168084 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:23:45.445279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.445244 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:23:45.451396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.451372 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.477095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.477067 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2a05c-openshift-default-dockercfg-lgjtt\"" Oct 30 01:23:45.482137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.482108 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:23:45.564219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564184 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564220 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564508 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564367 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564508 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564384 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564508 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564422 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564536 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564561 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564648 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpfr\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.564914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.564759 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.665845 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.665814 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.665850 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.665968 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.665994 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666044 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666118 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666153 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666232 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpfr\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666313 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666357 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666394 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666551 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666598 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.666891 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.666845 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.668681 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.668659 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.668822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.668803 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.683587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.683561 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.683587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.683577 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpfr\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:45.768527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:45.768431 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:46.101489 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:23:46.101458 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cf0ba4_b711_4a4a_902c_33bf164a5693.slice/crio-268992659d383976835dc27ad39eab9d9eb10c647779703cf65310687cbadcf3 WatchSource:0}: Error finding container 268992659d383976835dc27ad39eab9d9eb10c647779703cf65310687cbadcf3: Status 404 returned error can't find the container with id 268992659d383976835dc27ad39eab9d9eb10c647779703cf65310687cbadcf3 Oct 30 01:23:46.101920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.101884 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:23:46.104901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.104827 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:46.105042 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.104929 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:46.105042 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.104979 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:46.495744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.495702 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" event={"ID":"76cf0ba4-b711-4a4a-902c-33bf164a5693","Type":"ContainerStarted","Data":"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335"} Oct 30 01:23:46.496251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.495772 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" event={"ID":"76cf0ba4-b711-4a4a-902c-33bf164a5693","Type":"ContainerStarted","Data":"268992659d383976835dc27ad39eab9d9eb10c647779703cf65310687cbadcf3"} Oct 30 01:23:46.554665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.554603 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podStartSLOduration=1.554582176 podStartE2EDuration="1.554582176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:46.553686151 +0000 UTC m=+2417.396246745" watchObservedRunningTime="2026-10-30 01:23:46.554582176 +0000 UTC m=+2417.397142770" Oct 30 01:23:46.769111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:46.769008 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:47.720266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.720217 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:23:47.726882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.726761 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.742554 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.742514 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:23:47.811055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.811021 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:23:47.811760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.811708 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.812089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812065 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.812355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812336 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.812501 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812485 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.812646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812633 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.812790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812777 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.813013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.812994 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.813274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.813256 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.813545 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.813520 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwkp\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.814114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.814071 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 01:23:47.915146 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915111 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915320 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915199 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwkp\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915320 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915262 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915320 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915299 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915334 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915357 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915416 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915479 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915504 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.915647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.915610 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.916071 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.916043 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.916202 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.916167 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.916283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.916215 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.916419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.916397 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.917885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.917847 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.918348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.918327 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.956340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.956312 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:47.956484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:47.956464 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwkp\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:48.051614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.051529 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:48.428738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.428711 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:23:48.430125 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:23:48.430092 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0cbbd8_be34_49c9_8b06_645956c95542.slice/crio-dcb303ad69af39dd6baa0f89df71b54756a42fea90f69ffbdbf0c7bc0b6bbc28 WatchSource:0}: Error finding container dcb303ad69af39dd6baa0f89df71b54756a42fea90f69ffbdbf0c7bc0b6bbc28: Status 404 returned error can't find the container with id dcb303ad69af39dd6baa0f89df71b54756a42fea90f69ffbdbf0c7bc0b6bbc28 Oct 30 01:23:48.433175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.433115 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:48.433280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.433219 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:48.433280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.433252 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:23:48.511963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.511804 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" event={"ID":"7c0cbbd8-be34-49c9-8b06-645956c95542","Type":"ContainerStarted","Data":"dcb303ad69af39dd6baa0f89df71b54756a42fea90f69ffbdbf0c7bc0b6bbc28"} Oct 30 01:23:48.789941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.789350 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:23:48.789941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:48.789422 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 01:23:49.528308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:49.528269 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" event={"ID":"7c0cbbd8-be34-49c9-8b06-645956c95542","Type":"ContainerStarted","Data":"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255"} Oct 30 01:23:49.619885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:49.612517 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podStartSLOduration=2.6124883150000002 podStartE2EDuration="2.612488315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:49.604365579 +0000 UTC m=+2420.446926173" watchObservedRunningTime="2026-10-30 01:23:49.612488315 +0000 UTC m=+2420.455048926" Oct 30 01:23:49.774888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:49.771290 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:23:49.774888 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:49.771352 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 01:23:50.052423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:50.052391 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:50.053472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:50.053449 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:50.053561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:50.053500 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:50.775887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:50.771256 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:23:50.775887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:50.771325 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 01:23:51.052457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:51.052361 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:51.052457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:51.052434 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:51.786482 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:51.786445 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 01:23:51.786786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:51.786759 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 01:23:52.052932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:52.052829 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:52.053393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:52.052929 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:52.793921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:52.793883 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:52.794286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:52.794262 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:52.795395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:52.795362 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:23:53.052726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:53.052650 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:53.052906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:53.052729 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:54.052146 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:54.052111 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:54.052541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:54.052159 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:54.601309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:54.601272 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 01:23:54.601537 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:54.601512 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" containerID="cri-o://aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9" gracePeriod=30 Oct 30 01:23:55.052253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:55.052206 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:23:55.052668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:55.052272 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 01:23:55.433774 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:23:55.433736 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:23:56.056833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:56.056803 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:56.057266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:56.057125 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:56.057854 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:56.057838 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:23:58.743885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:58.738048 5548 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:58.743885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:23:58.738110 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:00.097075 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.097043 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 01:24:00.218031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.213370 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:24:00.218031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.214384 5548 cpu_manager.go:414] "containerMap: removing container" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" Oct 30 01:24:00.218031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.214404 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" Oct 30 01:24:00.218031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.214657 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerName="istio-proxy" Oct 30 01:24:00.234882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.231279 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:24:00.234882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.231404 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.249203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.249174 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.249820 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.249797 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250063 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250042 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250190 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250326 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250516 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250687 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.250815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.250801 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.251270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.251248 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\") pod \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\" (UID: \"0ccd4ec3-abd3-48fb-a7bf-313016e396a7\") " Oct 30 01:24:00.252513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.252490 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.254489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.253762 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.254489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.254194 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:00.255895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.255852 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.259832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.259810 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a6b650-openshift-default-dockercfg-h7k7v\"" Oct 30 01:24:00.260753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.260732 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.301050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.301019 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "kube-api-access-9dmq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:00.302258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.301892 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:00.303918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.303848 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:00.304132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.304108 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo" pod "0ccd4ec3-abd3-48fb-a7bf-313016e396a7" (UID: "0ccd4ec3-abd3-48fb-a7bf-313016e396a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:00.353253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353215 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.353451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353276 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.353451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353429 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxgs\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.353583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353551 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.353745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353721 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.353922 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.353846 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.354040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354016 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.354164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354053 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.354164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354113 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.354302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354279 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354308 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354323 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354339 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354352 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354364 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354376 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354420 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.354448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.354441 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmq4\" (UniqueName: \"kubernetes.io/projected/0ccd4ec3-abd3-48fb-a7bf-313016e396a7-kube-api-access-9dmq4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.456129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456070 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456108 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456145 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456185 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456226 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456298 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxgs\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456339 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456413 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.456682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456488 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.457636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.456908 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.457636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.457381 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.457636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.457555 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.457923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.457693 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.459143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.458011 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.460764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.460612 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.461925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.461898 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.475743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.475719 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.475923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.475899 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxgs\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\") pod \"e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.567655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.567574 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:00.631981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.631942 5548 generic.go:356] "Generic (PLEG): container finished" podID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" containerID="aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9" exitCode=0 Oct 30 01:24:00.632161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.631991 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" Oct 30 01:24:00.632161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.631999 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" event={"ID":"0ccd4ec3-abd3-48fb-a7bf-313016e396a7","Type":"ContainerDied","Data":"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9"} Oct 30 01:24:00.632161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.632055 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl" event={"ID":"0ccd4ec3-abd3-48fb-a7bf-313016e396a7","Type":"ContainerDied","Data":"eb978d07dc7f8e95543d07661f0af8a22f63fd2deabb57167ec2cc609dc282b9"} Oct 30 01:24:00.632161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.632090 5548 scope.go:122] "RemoveContainer" containerID="aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9" Oct 30 01:24:00.668210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.667669 5548 scope.go:122] "RemoveContainer" containerID="aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9" Oct 30 01:24:00.668210 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:24:00.668101 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9\": container with ID starting with aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9 not found: ID does not exist" containerID="aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9" Oct 30 01:24:00.668210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.668140 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9"} err="failed to get container status \"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9\": rpc error: code = NotFound desc = could not find container \"aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9\": container with ID starting with aa4264ce9d835d0c64312af0dd3a91d4d95906030f2286add90a0a16a3f08fa9 not found: ID does not exist" Oct 30 01:24:00.707096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.707069 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 01:24:00.713619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.713593 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-mdgsl"] Oct 30 01:24:00.906945 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.902580 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:24:00.906945 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:24:00.903937 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16525c9_524a_4635_8057_9712f693e64f.slice/crio-439a0f35bf1014700e3578265bab8fcc481f0749822623accd536a743c2bbeb6 WatchSource:0}: Error finding container 439a0f35bf1014700e3578265bab8fcc481f0749822623accd536a743c2bbeb6: Status 404 returned error can't find the container with id 439a0f35bf1014700e3578265bab8fcc481f0749822623accd536a743c2bbeb6 Oct 30 01:24:00.910066 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.910022 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:24:00.910169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.910115 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:24:00.910169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:00.910162 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:24:01.266884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:01.266802 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccd4ec3-abd3-48fb-a7bf-313016e396a7" path="/var/lib/kubelet/pods/0ccd4ec3-abd3-48fb-a7bf-313016e396a7/volumes" Oct 30 01:24:01.640734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:01.640642 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerStarted","Data":"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b"} Oct 30 01:24:01.640734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:01.640702 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerStarted","Data":"439a0f35bf1014700e3578265bab8fcc481f0749822623accd536a743c2bbeb6"} Oct 30 01:24:01.678551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:01.678499 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podStartSLOduration=2.678484793 podStartE2EDuration="2.678484793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:01.678238146 +0000 UTC m=+2432.520798797" watchObservedRunningTime="2026-10-30 01:24:01.678484793 +0000 UTC m=+2432.521045386" Oct 30 01:24:02.568592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:02.568559 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:02.574997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:02.574971 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:02.655936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:02.655906 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:02.657175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:02.657151 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:24:25.428585 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:24:25.428541 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:24:39.884947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:39.884910 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 01:24:39.885496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:39.885133 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" containerID="cri-o://a257fbf5b8ecfc368000118c8e99b6142e503719048fab8cc50ff9f89479106e" gracePeriod=30 Oct 30 01:24:43.039839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:43.039801 5548 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:24:43.040273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:43.039896 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:24:45.128913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.128830 5548 generic.go:356] "Generic (PLEG): container finished" podID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerID="a257fbf5b8ecfc368000118c8e99b6142e503719048fab8cc50ff9f89479106e" exitCode=0 Oct 30 01:24:45.129468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.128913 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" event={"ID":"63acbafe-8598-4c0b-a81d-8ba6d0f97799","Type":"ContainerDied","Data":"a257fbf5b8ecfc368000118c8e99b6142e503719048fab8cc50ff9f89479106e"} Oct 30 01:24:45.172144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.172120 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 01:24:45.243465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.243430 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:24:45.244595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.244572 5548 cpu_manager.go:414] "containerMap: removing container" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" Oct 30 01:24:45.244661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.244600 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" Oct 30 01:24:45.244813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.244800 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" containerName="istio-proxy" Oct 30 01:24:45.249224 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.249201 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.262095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.262069 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2a05c\"" Oct 30 01:24:45.262270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.262248 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2a05c-dockercfg-zltrz\"" Oct 30 01:24:45.263785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.263762 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264059 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264040 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264226 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264105 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.264416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264319 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.264576 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264551 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264610 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264668 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264796 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.264838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264828 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.265018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.264957 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\") pod \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\" (UID: \"63acbafe-8598-4c0b-a81d-8ba6d0f97799\") " Oct 30 01:24:45.265373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.265347 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc4n\" (UniqueName: \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.265722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.265695 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.265923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.265906 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.265991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.265926 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.266152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.266136 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.267551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.267252 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:45.267551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.267282 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.267702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.267643 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:45.271389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.271361 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:45.273428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.273401 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "kube-api-access-5lcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:45.274457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.274412 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo" pod "63acbafe-8598-4c0b-a81d-8ba6d0f97799" (UID: "63acbafe-8598-4c0b-a81d-8ba6d0f97799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:24:45.300004 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.299980 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:24:45.367009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.366975 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc4n\" (UniqueName: \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.367191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367133 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.367289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367239 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367253 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lcs2\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-kube-api-access-5lcs2\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367262 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367275 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367286 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367294 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.367498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.367308 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63acbafe-8598-4c0b-a81d-8ba6d0f97799-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:24:45.368124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.368084 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.384526 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.384497 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc4n\" (UniqueName: \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\") pod \"payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.568752 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.568661 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:45.879002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:45.878974 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:24:45.879380 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:24:45.879358 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a977fe0_6cb2_4092_a87a_8610752496f0.slice/crio-18d1abe5515497e12adc191908fc891a2f00cc0abf77a49084c0e55ce410cfee WatchSource:0}: Error finding container 18d1abe5515497e12adc191908fc891a2f00cc0abf77a49084c0e55ce410cfee: Status 404 returned error can't find the container with id 18d1abe5515497e12adc191908fc891a2f00cc0abf77a49084c0e55ce410cfee Oct 30 01:24:46.140018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.139942 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" Oct 30 01:24:46.140018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.139950 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6" event={"ID":"63acbafe-8598-4c0b-a81d-8ba6d0f97799","Type":"ContainerDied","Data":"8372e9f50d22014b4d4a16123c5f84f73fc13e7923a8a4d9002993907144f920"} Oct 30 01:24:46.140561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.140020 5548 scope.go:122] "RemoveContainer" containerID="a257fbf5b8ecfc368000118c8e99b6142e503719048fab8cc50ff9f89479106e" Oct 30 01:24:46.142154 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.142129 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" event={"ID":"7a977fe0-6cb2-4092-a87a-8610752496f0","Type":"ContainerStarted","Data":"32ea92109f47d1953916fef91de332e27320e9c0ec39cfbbbb09b9d7c02085c0"} Oct 30 01:24:46.142283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.142179 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" event={"ID":"7a977fe0-6cb2-4092-a87a-8610752496f0","Type":"ContainerStarted","Data":"18d1abe5515497e12adc191908fc891a2f00cc0abf77a49084c0e55ce410cfee"} Oct 30 01:24:46.142283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.142208 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:24:46.185685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.185648 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 01:24:46.196024 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.195992 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-64wz6"] Oct 30 01:24:46.222743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:46.222673 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" podStartSLOduration=26.222652746 podStartE2EDuration="26.222652746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:46.219177989 +0000 UTC m=+2477.061738609" watchObservedRunningTime="2026-10-30 01:24:46.222652746 +0000 UTC m=+2477.065213340" Oct 30 01:24:47.267165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:47.267134 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63acbafe-8598-4c0b-a81d-8ba6d0f97799" path="/var/lib/kubelet/pods/63acbafe-8598-4c0b-a81d-8ba6d0f97799/volumes" Oct 30 01:24:55.429735 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:24:55.429694 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:24:57.155412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:24:57.155383 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:25:04.236915 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:04.236886 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj_7a977fe0-6cb2-4092-a87a-8610752496f0/payload-processing/0.log" Oct 30 01:25:25.432495 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:25:25.432448 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:25:26.540278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.540244 5548 generic.go:356] "Generic (PLEG): container finished" podID="a16525c9-524a-4635-8057-9712f693e64f" containerID="c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b" exitCode=0 Oct 30 01:25:26.540803 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.540299 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerDied","Data":"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b"} Oct 30 01:25:26.540944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.540925 5548 scope.go:122] "RemoveContainer" containerID="c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b" Oct 30 01:25:26.542893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.542829 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:25:26.543029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.542941 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:25:26.543029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:26.542997 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:25:27.550908 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:27.550839 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerStarted","Data":"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a"} Oct 30 01:25:27.579279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:27.579246 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:25:27.579637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:27.579612 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:27.579825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:27.579801 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:28.588304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:28.588264 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:28.588963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:28.588926 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:29.592683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:29.585599 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:29.592683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:29.585687 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:30.593056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:30.593018 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:30.593666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:30.593634 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:31.582280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:31.582242 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:31.582573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:31.582543 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:32.576885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:32.571263 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 01:25:32.576885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:32.571324 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 01:25:32.655757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:32.655722 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:25:33.573790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:33.573757 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:25:33.618008 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:33.617980 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:25:38.636079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:38.636043 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:25:38.636561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:38.636288 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" podUID="90826748-a433-453c-9bfb-a0a26ac1b4b9" containerName="maas-api" containerID="cri-o://7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb" gracePeriod=30 Oct 30 01:25:38.784477 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:38.784445 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:25:38.784674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:38.784648 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" podUID="2376c94c-298d-477c-85b3-9201f67def0b" containerName="payload-processing" containerID="cri-o://5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7" gracePeriod=30 Oct 30 01:25:39.038529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.038172 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:25:39.142064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.142001 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:25:39.143583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.143534 5548 cpu_manager.go:414] "containerMap: removing container" podUID="90826748-a433-453c-9bfb-a0a26ac1b4b9" containerName="maas-api" Oct 30 01:25:39.143765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.143750 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90826748-a433-453c-9bfb-a0a26ac1b4b9" containerName="maas-api" Oct 30 01:25:39.144069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.144050 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="90826748-a433-453c-9bfb-a0a26ac1b4b9" containerName="maas-api" Oct 30 01:25:39.153731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.153707 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.198383 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.197393 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f0263\"" Oct 30 01:25:39.198383 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.197521 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f0263-dockercfg-qrb66\"" Oct 30 01:25:39.217790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.209924 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.219670 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\" (UniqueName: \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\") pod \"90826748-a433-453c-9bfb-a0a26ac1b4b9\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.219753 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\") pod \"90826748-a433-453c-9bfb-a0a26ac1b4b9\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.219834 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") pod \"90826748-a433-453c-9bfb-a0a26ac1b4b9\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.219976 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\") pod \"90826748-a433-453c-9bfb-a0a26ac1b4b9\" (UID: \"90826748-a433-453c-9bfb-a0a26ac1b4b9\") " Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.220651 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.250290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.220740 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcf7\" (UniqueName: \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.292786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.285948 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca" pod "90826748-a433-453c-9bfb-a0a26ac1b4b9" (UID: "90826748-a433-453c-9bfb-a0a26ac1b4b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:39.317951 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.294065 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs" pod "90826748-a433-453c-9bfb-a0a26ac1b4b9" (UID: "90826748-a433-453c-9bfb-a0a26ac1b4b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:39.325972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.325261 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb" pod "90826748-a433-453c-9bfb-a0a26ac1b4b9" (UID: "90826748-a433-453c-9bfb-a0a26ac1b4b9"). InnerVolumeSpecName "kube-api-access-tv9lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:39.327506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.327450 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls" pod "90826748-a433-453c-9bfb-a0a26ac1b4b9" (UID: "90826748-a433-453c-9bfb-a0a26ac1b4b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.327172 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglmd\" (UniqueName: \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.331947 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.331995 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.332083 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcf7\" (UniqueName: \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.332402 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9lb\" (UniqueName: \"kubernetes.io/projected/90826748-a433-453c-9bfb-a0a26ac1b4b9-kube-api-access-tv9lb\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.332420 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90826748-a433-453c-9bfb-a0a26ac1b4b9-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.332433 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.332445 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90826748-a433-453c-9bfb-a0a26ac1b4b9-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.333404 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.333563 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.346553 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a6b650-dockercfg-4z8zw\"" Oct 30 01:25:39.354049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.347445 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a6b650\"" Oct 30 01:25:39.365779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.361180 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcf7\" (UniqueName: \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\") pod \"payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.365779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.363922 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:25:39.375913 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.375835 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:25:39.383903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.383587 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:25:39.433564 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.433532 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:25:39.434277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.434259 5548 cpu_manager.go:414] "containerMap: removing container" podUID="2376c94c-298d-477c-85b3-9201f67def0b" containerName="payload-processing" Oct 30 01:25:39.434277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.434278 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2376c94c-298d-477c-85b3-9201f67def0b" containerName="payload-processing" Oct 30 01:25:39.434438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.434427 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="2376c94c-298d-477c-85b3-9201f67def0b" containerName="payload-processing" Oct 30 01:25:39.437272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.437231 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglmd\" (UniqueName: \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.437438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.437287 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.439698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.439670 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.444015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.443991 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.461191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.461164 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglmd\" (UniqueName: \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\") pod \"payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.463387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.463358 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:25:39.538916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.538855 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\") pod \"2376c94c-298d-477c-85b3-9201f67def0b\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " Oct 30 01:25:39.539214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.539186 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\" (UniqueName: \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\") pod \"2376c94c-298d-477c-85b3-9201f67def0b\" (UID: \"2376c94c-298d-477c-85b3-9201f67def0b\") " Oct 30 01:25:39.539285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.539234 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume" pod "2376c94c-298d-477c-85b3-9201f67def0b" (UID: "2376c94c-298d-477c-85b3-9201f67def0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:39.539607 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.539577 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v484\" (UniqueName: \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.539752 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.539628 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.540061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.540030 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2376c94c-298d-477c-85b3-9201f67def0b-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.541482 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.541463 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj" pod "2376c94c-298d-477c-85b3-9201f67def0b" (UID: "2376c94c-298d-477c-85b3-9201f67def0b"). InnerVolumeSpecName "kube-api-access-958sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:39.641226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.641186 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v484\" (UniqueName: \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.641653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.641238 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.641653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.641397 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958sj\" (UniqueName: \"kubernetes.io/projected/2376c94c-298d-477c-85b3-9201f67def0b-kube-api-access-958sj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:25:39.642007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.641986 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.644308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.643660 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:39.658557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.658529 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v484\" (UniqueName: \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\") pod \"payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.678854 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.678830 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:39.687278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.687248 5548 generic.go:356] "Generic (PLEG): container finished" podID="2376c94c-298d-477c-85b3-9201f67def0b" containerID="5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7" exitCode=0 Oct 30 01:25:39.687431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.687321 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" Oct 30 01:25:39.687431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.687331 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" event={"ID":"2376c94c-298d-477c-85b3-9201f67def0b","Type":"ContainerDied","Data":"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7"} Oct 30 01:25:39.687431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.687394 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs" event={"ID":"2376c94c-298d-477c-85b3-9201f67def0b","Type":"ContainerDied","Data":"9cf1a737cb692ee1684d88f1d32d846ff29285bb39a9590c47d7a00c1bb5319f"} Oct 30 01:25:39.687431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.687424 5548 scope.go:122] "RemoveContainer" containerID="5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7" Oct 30 01:25:39.691117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.689198 5548 generic.go:356] "Generic (PLEG): container finished" podID="90826748-a433-453c-9bfb-a0a26ac1b4b9" containerID="7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb" exitCode=0 Oct 30 01:25:39.691117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.689325 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" event={"ID":"90826748-a433-453c-9bfb-a0a26ac1b4b9","Type":"ContainerDied","Data":"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb"} Oct 30 01:25:39.691117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.689370 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" event={"ID":"90826748-a433-453c-9bfb-a0a26ac1b4b9","Type":"ContainerDied","Data":"7388744172c6aff135f36f2adddeba515c0c77ac3828b5ba8f40fc8724fb0ffb"} Oct 30 01:25:39.691117 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.689440 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v" Oct 30 01:25:39.752828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.752723 5548 scope.go:122] "RemoveContainer" containerID="5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7" Oct 30 01:25:39.753230 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:25:39.753080 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7\": container with ID starting with 5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7 not found: ID does not exist" containerID="5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7" Oct 30 01:25:39.753230 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.753117 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7"} err="failed to get container status \"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7\": rpc error: code = NotFound desc = could not find container \"5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7\": container with ID starting with 5e668b26a616322a1bb525b591a2ceb4eea8d5fc70f3a5bcdc0dcfc965f092c7 not found: ID does not exist" Oct 30 01:25:39.753230 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.753141 5548 scope.go:122] "RemoveContainer" containerID="7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb" Oct 30 01:25:39.772759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.772526 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:39.777711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.777660 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:25:39.785036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.785007 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d32079-65bd949f84-bxcqs"] Oct 30 01:25:39.816697 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.816626 5548 scope.go:122] "RemoveContainer" containerID="7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb" Oct 30 01:25:39.817324 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:25:39.817143 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb\": container with ID starting with 7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb not found: ID does not exist" containerID="7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb" Oct 30 01:25:39.817324 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.817178 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb"} err="failed to get container status \"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb\": rpc error: code = NotFound desc = could not find container \"7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb\": container with ID starting with 7def582f186ebd47be3a0573a439ba3fd4d0f06d7f52fcfabaafe0d83f23e3cb not found: ID does not exist" Oct 30 01:25:39.829917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.829893 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:25:39.843905 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.843719 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d32079-766df59dc4-bff8v"] Oct 30 01:25:39.980490 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:39.980458 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:25:39.980637 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:25:39.980612 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977c9c97_6d79_486a_8402_d16f16925b1c.slice/crio-bbc5372aefb355d8fe15d38b88be801a9173aac34d165be6b7382ae7b1fd7176 WatchSource:0}: Error finding container bbc5372aefb355d8fe15d38b88be801a9173aac34d165be6b7382ae7b1fd7176: Status 404 returned error can't find the container with id bbc5372aefb355d8fe15d38b88be801a9173aac34d165be6b7382ae7b1fd7176 Oct 30 01:25:40.068977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.068938 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:25:40.069330 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:25:40.069307 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f01b7d5_f187_4123_9b48_3698e7f94470.slice/crio-ce3275f1efbe46f51414f62f5862b0ed8943a28f379591c899fb6062ba028252 WatchSource:0}: Error finding container ce3275f1efbe46f51414f62f5862b0ed8943a28f379591c899fb6062ba028252: Status 404 returned error can't find the container with id ce3275f1efbe46f51414f62f5862b0ed8943a28f379591c899fb6062ba028252 Oct 30 01:25:40.071272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.071253 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:25:40.141314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.141270 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:25:40.142140 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:25:40.142111 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7070c53f_fbfd_4a6a_8e44_fac0064c300c.slice/crio-89601834ac02d4219df35aaff268c33b11193322d0ef55cf5605a59249a20d8a WatchSource:0}: Error finding container 89601834ac02d4219df35aaff268c33b11193322d0ef55cf5605a59249a20d8a: Status 404 returned error can't find the container with id 89601834ac02d4219df35aaff268c33b11193322d0ef55cf5605a59249a20d8a Oct 30 01:25:40.718308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.718268 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" event={"ID":"977c9c97-6d79-486a-8402-d16f16925b1c","Type":"ContainerStarted","Data":"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a"} Oct 30 01:25:40.718853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.718340 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" event={"ID":"977c9c97-6d79-486a-8402-d16f16925b1c","Type":"ContainerStarted","Data":"bbc5372aefb355d8fe15d38b88be801a9173aac34d165be6b7382ae7b1fd7176"} Oct 30 01:25:40.718853 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.718830 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:40.722792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.722759 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" event={"ID":"7070c53f-fbfd-4a6a-8e44-fac0064c300c","Type":"ContainerStarted","Data":"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8"} Oct 30 01:25:40.722964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.722818 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:40.722964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.722841 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" event={"ID":"7070c53f-fbfd-4a6a-8e44-fac0064c300c","Type":"ContainerStarted","Data":"89601834ac02d4219df35aaff268c33b11193322d0ef55cf5605a59249a20d8a"} Oct 30 01:25:40.724373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.724346 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" event={"ID":"4f01b7d5-f187-4123-9b48-3698e7f94470","Type":"ContainerStarted","Data":"ce3275f1efbe46f51414f62f5862b0ed8943a28f379591c899fb6062ba028252"} Oct 30 01:25:40.755742 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.755694 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" podStartSLOduration=26.755678356 podStartE2EDuration="26.755678356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:40.752758979 +0000 UTC m=+2531.595319572" watchObservedRunningTime="2026-10-30 01:25:40.755678356 +0000 UTC m=+2531.598238949" Oct 30 01:25:40.791294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:40.791236 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" podStartSLOduration=25.791216627 podStartE2EDuration="25.791216627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:40.786202019 +0000 UTC m=+2531.628762637" watchObservedRunningTime="2026-10-30 01:25:40.791216627 +0000 UTC m=+2531.633777221" Oct 30 01:25:41.267781 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:41.267747 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2376c94c-298d-477c-85b3-9201f67def0b" path="/var/lib/kubelet/pods/2376c94c-298d-477c-85b3-9201f67def0b/volumes" Oct 30 01:25:41.268527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:41.268499 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90826748-a433-453c-9bfb-a0a26ac1b4b9" path="/var/lib/kubelet/pods/90826748-a433-453c-9bfb-a0a26ac1b4b9/volumes" Oct 30 01:25:42.744919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:42.744847 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" event={"ID":"4f01b7d5-f187-4123-9b48-3698e7f94470","Type":"ContainerStarted","Data":"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b"} Oct 30 01:25:42.745345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:42.744943 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:42.784827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:42.784776 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" podStartSLOduration=94.675943088 podStartE2EDuration="1m36.7847623s" totalImagesPullingTime="2.108819212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:42.781314837 +0000 UTC m=+2533.623875430" watchObservedRunningTime="2026-10-30 01:25:42.7847623 +0000 UTC m=+2533.627322894" Oct 30 01:25:51.737695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:51.737664 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:25:51.738132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:51.737736 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:25:53.059184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:53.059153 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj_3bdec348-3d68-454d-9c07-18a6e0d90a49/payload-processing/0.log" Oct 30 01:25:53.644621 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:53.644586 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw_7070c53f-fbfd-4a6a-8e44-fac0064c300c/payload-processing/0.log" Oct 30 01:25:53.756139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:25:53.756111 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:25:55.431009 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:25:55.430961 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:26:25.430557 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:26:25.430515 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:26:38.622461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.622426 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:26:38.623158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.622652 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" podUID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" containerName="maas-api" containerID="cri-o://c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43" gracePeriod=30 Oct 30 01:26:38.634947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.634915 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh"] Oct 30 01:26:38.644420 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.644394 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.661069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.659800 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:26:38.661194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.661147 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:26:38.668946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.665742 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh"] Oct 30 01:26:38.686259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.686162 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.686420 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.686395 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzrb\" (UniqueName: \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.686687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.686648 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.776594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.776550 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:26:38.776847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.776818 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" podUID="7a977fe0-6cb2-4092-a87a-8610752496f0" containerName="payload-processing" containerID="cri-o://32ea92109f47d1953916fef91de332e27320e9c0ec39cfbbbb09b9d7c02085c0" gracePeriod=30 Oct 30 01:26:38.788568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.788535 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.788749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.788690 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.788982 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.788956 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzrb\" (UniqueName: \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.789605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.789556 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.806572 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.806536 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzrb\" (UniqueName: \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:38.806710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:38.806574 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e7a0ac-8t8nh\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:39.026164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.026017 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:39.237587 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.237547 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:26:39.348708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.348666 5548 generic.go:356] "Generic (PLEG): container finished" podID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" containerID="c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43" exitCode=0 Oct 30 01:26:39.349460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.348949 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" event={"ID":"b91fc1d5-3914-4e9f-9396-fa54b88f0065","Type":"ContainerDied","Data":"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43"} Oct 30 01:26:39.349460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.349135 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" event={"ID":"b91fc1d5-3914-4e9f-9396-fa54b88f0065","Type":"ContainerDied","Data":"bdfda1cc89abc73624caf827119a3e9164256b0767404c25fefb3c583494e178"} Oct 30 01:26:39.349460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.349174 5548 scope.go:122] "RemoveContainer" containerID="c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43" Oct 30 01:26:39.349460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.348994 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr" Oct 30 01:26:39.354507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.354480 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:26:39.361346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.361281 5548 generic.go:356] "Generic (PLEG): container finished" podID="7a977fe0-6cb2-4092-a87a-8610752496f0" containerID="32ea92109f47d1953916fef91de332e27320e9c0ec39cfbbbb09b9d7c02085c0" exitCode=0 Oct 30 01:26:39.362373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.361791 5548 cpu_manager.go:414] "containerMap: removing container" podUID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" containerName="maas-api" Oct 30 01:26:39.362373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.361814 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" containerName="maas-api" Oct 30 01:26:39.362373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.362186 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" containerName="maas-api" Oct 30 01:26:39.380652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.380624 5548 scope.go:122] "RemoveContainer" containerID="c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43" Oct 30 01:26:39.389775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.389720 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" event={"ID":"7a977fe0-6cb2-4092-a87a-8610752496f0","Type":"ContainerDied","Data":"32ea92109f47d1953916fef91de332e27320e9c0ec39cfbbbb09b9d7c02085c0"} Oct 30 01:26:39.390123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.390088 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:26:39.391061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.390297 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:26:39.391061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.390423 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.407137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.407081 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-29bb56-openshift-default-dockercfg-7rd5k\"" Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:26:39.410617 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43\": container with ID starting with c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43 not found: ID does not exist" containerID="c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43" Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.410662 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43"} err="failed to get container status \"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43\": rpc error: code = NotFound desc = could not find container \"c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43\": container with ID starting with c525815a137314236938ccb6d87fd8579faf8ce84f6f0692a06da17e33171f43 not found: ID does not exist" Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.420083 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\") pod \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.420185 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") pod \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.420510 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\" (UniqueName: \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\") pod \"7a977fe0-6cb2-4092-a87a-8610752496f0\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.421367 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\") pod \"7a977fe0-6cb2-4092-a87a-8610752496f0\" (UID: \"7a977fe0-6cb2-4092-a87a-8610752496f0\") " Oct 30 01:26:39.422116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.421446 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\" (UniqueName: \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\") pod \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " Oct 30 01:26:39.422669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.422240 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume" pod "7a977fe0-6cb2-4092-a87a-8610752496f0" (UID: "7a977fe0-6cb2-4092-a87a-8610752496f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:39.422669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.422413 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\") pod \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\" (UID: \"b91fc1d5-3914-4e9f-9396-fa54b88f0065\") " Oct 30 01:26:39.423118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423073 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca" pod "b91fc1d5-3914-4e9f-9396-fa54b88f0065" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423633 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423736 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423809 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423841 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.423930 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgdg\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.424006 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.424188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.424032 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.426078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.425704 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.426078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.425751 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.426078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.425941 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a977fe0-6cb2-4092-a87a-8610752496f0-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.426078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.425966 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91fc1d5-3914-4e9f-9396-fa54b88f0065-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.444181 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.443569 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw" pod "b91fc1d5-3914-4e9f-9396-fa54b88f0065" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065"). InnerVolumeSpecName "kube-api-access-6l9qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:39.447618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.446430 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n" pod "7a977fe0-6cb2-4092-a87a-8610752496f0" (UID: "7a977fe0-6cb2-4092-a87a-8610752496f0"). InnerVolumeSpecName "kube-api-access-xrc4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:39.458076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.458041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs" pod "b91fc1d5-3914-4e9f-9396-fa54b88f0065" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:39.460350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.460316 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls" pod "b91fc1d5-3914-4e9f-9396-fa54b88f0065" (UID: "b91fc1d5-3914-4e9f-9396-fa54b88f0065"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:39.503343 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.503249 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:26:39.504815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.504786 5548 cpu_manager.go:414] "containerMap: removing container" podUID="7a977fe0-6cb2-4092-a87a-8610752496f0" containerName="payload-processing" Oct 30 01:26:39.505673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.505651 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a977fe0-6cb2-4092-a87a-8610752496f0" containerName="payload-processing" Oct 30 01:26:39.506096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.506076 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a977fe0-6cb2-4092-a87a-8610752496f0" containerName="payload-processing" Oct 30 01:26:39.512776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.512746 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:26:39.512932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.512885 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.525902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.525831 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-368bbb-serving-cert\"" Oct 30 01:26:39.528271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528237 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpjd\" (UniqueName: \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.528406 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528317 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgdg\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528406 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528383 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528409 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528517 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528553 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528604 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.528716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528699 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.528754 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528739 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.528814 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528798 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.528937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.528913 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.529178 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529159 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.529316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529301 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.529512 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529495 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrc4n\" (UniqueName: \"kubernetes.io/projected/7a977fe0-6cb2-4092-a87a-8610752496f0-kube-api-access-xrc4n\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.529625 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529612 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l9qw\" (UniqueName: \"kubernetes.io/projected/b91fc1d5-3914-4e9f-9396-fa54b88f0065-kube-api-access-6l9qw\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.529730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529718 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.529824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.529814 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91fc1d5-3914-4e9f-9396-fa54b88f0065-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:39.536402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.536365 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.537334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.537311 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.539459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.537428 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.539613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.537673 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.539713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.537940 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.542577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.542553 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.545570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.545544 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.568776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.568738 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgdg\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.585492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.585458 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\") pod \"e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.631598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.631557 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpjd\" (UniqueName: \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.632152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.631748 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.632152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.631834 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.632152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.631918 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.632672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.632605 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.637335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.636105 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.637335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.636754 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.638902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.638835 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh"] Oct 30 01:26:39.639278 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:26:39.639253 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4553c4_9f56_4590_83ec_a06ff2d0050e.slice/crio-5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892 WatchSource:0}: Error finding container 5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892: Status 404 returned error can't find the container with id 5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892 Oct 30 01:26:39.659225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.659195 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpjd\" (UniqueName: \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\") pod \"maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:39.743090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.743053 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:26:39.746532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.746493 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:39.749191 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.749162 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2a05c-5969f9cd79-vj6gr"] Oct 30 01:26:39.843590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:39.843560 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:40.078573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.078545 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:26:40.078674 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:26:40.078659 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8865e1c8_27b0_4549_b818_2068e42c7cbb.slice/crio-8beb44ddbdd5f5d6b53b2d5b13e40a32e0c4c4a15a3769f5554cb7db6cc496d5 WatchSource:0}: Error finding container 8beb44ddbdd5f5d6b53b2d5b13e40a32e0c4c4a15a3769f5554cb7db6cc496d5: Status 404 returned error can't find the container with id 8beb44ddbdd5f5d6b53b2d5b13e40a32e0c4c4a15a3769f5554cb7db6cc496d5 Oct 30 01:26:40.082874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.082813 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:26:40.082960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.082925 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:26:40.083001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.082978 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:26:40.171261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.171231 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:26:40.176609 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:26:40.176568 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecfccee_49fb_43d2_8382_d9f4880f78db.slice/crio-64838f5927b21b7d6eda783d7a409b130fdc48bf79b73a0cb0d33c9186108bd6 WatchSource:0}: Error finding container 64838f5927b21b7d6eda783d7a409b130fdc48bf79b73a0cb0d33c9186108bd6: Status 404 returned error can't find the container with id 64838f5927b21b7d6eda783d7a409b130fdc48bf79b73a0cb0d33c9186108bd6 Oct 30 01:26:40.371225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.371139 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" Oct 30 01:26:40.371401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.371140 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj" event={"ID":"7a977fe0-6cb2-4092-a87a-8610752496f0","Type":"ContainerDied","Data":"18d1abe5515497e12adc191908fc891a2f00cc0abf77a49084c0e55ce410cfee"} Oct 30 01:26:40.371401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.371310 5548 scope.go:122] "RemoveContainer" containerID="32ea92109f47d1953916fef91de332e27320e9c0ec39cfbbbb09b9d7c02085c0" Oct 30 01:26:40.372796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.372765 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" event={"ID":"fecfccee-49fb-43d2-8382-d9f4880f78db","Type":"ContainerStarted","Data":"64838f5927b21b7d6eda783d7a409b130fdc48bf79b73a0cb0d33c9186108bd6"} Oct 30 01:26:40.374739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.374719 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerStarted","Data":"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a"} Oct 30 01:26:40.374901 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.374762 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerStarted","Data":"8beb44ddbdd5f5d6b53b2d5b13e40a32e0c4c4a15a3769f5554cb7db6cc496d5"} Oct 30 01:26:40.377923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.377897 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" event={"ID":"aa4553c4-9f56-4590-83ec-a06ff2d0050e","Type":"ContainerStarted","Data":"181f251749ea4e5f70bfa820c93959af8879eaffe31d2b8569c25fecee6a2087"} Oct 30 01:26:40.378051 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.377945 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" event={"ID":"aa4553c4-9f56-4590-83ec-a06ff2d0050e","Type":"ContainerStarted","Data":"5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892"} Oct 30 01:26:40.442684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.442652 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:26:40.455819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.455787 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2a05c-5d998c694b-6lglj"] Oct 30 01:26:40.511493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.508352 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" podStartSLOduration=2.508329226 podStartE2EDuration="2.508329226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:40.505541394 +0000 UTC m=+2591.348101988" watchObservedRunningTime="2026-10-30 01:26:40.508329226 +0000 UTC m=+2591.350889823" Oct 30 01:26:40.546516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.546467 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podStartSLOduration=16.546452638 podStartE2EDuration="16.546452638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:40.543425184 +0000 UTC m=+2591.385985779" watchObservedRunningTime="2026-10-30 01:26:40.546452638 +0000 UTC m=+2591.389013231" Oct 30 01:26:40.747592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:40.747557 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:41.266923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.266893 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a977fe0-6cb2-4092-a87a-8610752496f0" path="/var/lib/kubelet/pods/7a977fe0-6cb2-4092-a87a-8610752496f0/volumes" Oct 30 01:26:41.267392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.267373 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91fc1d5-3914-4e9f-9396-fa54b88f0065" path="/var/lib/kubelet/pods/b91fc1d5-3914-4e9f-9396-fa54b88f0065/volumes" Oct 30 01:26:41.388135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.388097 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" event={"ID":"fecfccee-49fb-43d2-8382-d9f4880f78db","Type":"ContainerStarted","Data":"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997"} Oct 30 01:26:41.388341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.388183 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:41.389759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.389730 5548 generic.go:356] "Generic (PLEG): container finished" podID="aa4553c4-9f56-4590-83ec-a06ff2d0050e" containerID="181f251749ea4e5f70bfa820c93959af8879eaffe31d2b8569c25fecee6a2087" exitCode=0 Oct 30 01:26:41.389949 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.389815 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" event={"ID":"aa4553c4-9f56-4590-83ec-a06ff2d0050e","Type":"ContainerDied","Data":"181f251749ea4e5f70bfa820c93959af8879eaffe31d2b8569c25fecee6a2087"} Oct 30 01:26:41.439493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.430691 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" podStartSLOduration=16.152342494 podStartE2EDuration="16.430668128s" totalImagesPullingTime="278.325634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:41.428238512 +0000 UTC m=+2592.270799091" watchObservedRunningTime="2026-10-30 01:26:41.430668128 +0000 UTC m=+2592.273228723" Oct 30 01:26:41.753416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:41.753384 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:42.397717 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.397175 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:42.398695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.398671 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:26:42.594032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.594004 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:42.687188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.687104 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\" (UniqueName: \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\") pod \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " Oct 30 01:26:42.687188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.687153 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\") pod \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " Oct 30 01:26:42.687439 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.687394 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\") pod \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\" (UID: \"aa4553c4-9f56-4590-83ec-a06ff2d0050e\") " Oct 30 01:26:42.687945 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.687849 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca" pod "aa4553c4-9f56-4590-83ec-a06ff2d0050e" (UID: "aa4553c4-9f56-4590-83ec-a06ff2d0050e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:42.688175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.688155 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:42.689574 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.689547 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca" pod "aa4553c4-9f56-4590-83ec-a06ff2d0050e" (UID: "aa4553c4-9f56-4590-83ec-a06ff2d0050e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:42.689695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.689600 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb" pod "aa4553c4-9f56-4590-83ec-a06ff2d0050e" (UID: "aa4553c4-9f56-4590-83ec-a06ff2d0050e"). InnerVolumeSpecName "kube-api-access-jmzrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:42.789462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.789433 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzrb\" (UniqueName: \"kubernetes.io/projected/aa4553c4-9f56-4590-83ec-a06ff2d0050e-kube-api-access-jmzrb\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:42.789462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:42.789463 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4553c4-9f56-4590-83ec-a06ff2d0050e-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:43.411900 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:43.411820 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" Oct 30 01:26:43.412083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:43.411972 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh" event={"ID":"aa4553c4-9f56-4590-83ec-a06ff2d0050e","Type":"ContainerDied","Data":"5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892"} Oct 30 01:26:43.412083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:43.412029 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5624335bc7eee89d924e33010bf44b7e0e4da8ea0c8de33c61f1c76b0d2a0892" Oct 30 01:26:46.501167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:46.500977 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:26:46.501675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:46.501263 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" containerID="cri-o://beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335" gracePeriod=30 Oct 30 01:26:47.404773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:47.404737 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:26:51.839378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.839347 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:26:51.912045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.911998 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:26:51.914717 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.914686 5548 cpu_manager.go:414] "containerMap: removing container" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" Oct 30 01:26:51.914717 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.914716 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" Oct 30 01:26:51.914953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.914796 5548 cpu_manager.go:414] "containerMap: removing container" podUID="aa4553c4-9f56-4590-83ec-a06ff2d0050e" containerName="revoke-keys" Oct 30 01:26:51.914953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.914807 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4553c4-9f56-4590-83ec-a06ff2d0050e" containerName="revoke-keys" Oct 30 01:26:51.915510 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.915487 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerName="istio-proxy" Oct 30 01:26:51.915615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.915517 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4553c4-9f56-4590-83ec-a06ff2d0050e" containerName="revoke-keys" Oct 30 01:26:51.924968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.924942 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:51.930061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.930035 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:26:51.943499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943471 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.943608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943550 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.943608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943590 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.943739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943694 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.943796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943756 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.943940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943916 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.944077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943955 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.944077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.943986 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.944077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944075 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.944281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944105 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.944281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944122 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\") pod \"76cf0ba4-b711-4a4a-902c-33bf164a5693\" (UID: \"76cf0ba4-b711-4a4a-902c-33bf164a5693\") " Oct 30 01:26:51.944281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944250 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.944470 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944446 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:51.944629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944609 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:51.944705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944684 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:51.944768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944709 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:51.944882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.944846 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4b76\" (UniqueName: \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:51.945047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.945019 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.945155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.945114 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.945155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.945139 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.945155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.945153 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.945334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.945226 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf0ba4-b711-4a4a-902c-33bf164a5693-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.946449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.946418 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "kube-api-access-ccpfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:51.946797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.946769 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.947265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.947243 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:51.947962 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.947944 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token" pod "76cf0ba4-b711-4a4a-902c-33bf164a5693" (UID: "76cf0ba4-b711-4a4a-902c-33bf164a5693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:51.987651 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:51.987626 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a6b650-serving-cert\"" Oct 30 01:26:52.046777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.046740 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.046979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.046844 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4b76\" (UniqueName: \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.047072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047056 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.047185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047172 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.047292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047279 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpfr\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-kube-api-access-ccpfr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:52.047360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047298 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:52.047360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047313 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:52.047360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047327 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:52.047360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047341 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf0ba4-b711-4a4a-902c-33bf164a5693-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:26:52.047667 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.047644 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.049503 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.049484 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.049555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.049525 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.063602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.063548 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4b76\" (UniqueName: \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\") pod \"maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.247009 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.246970 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:52.553018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.552986 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:26:52.553765 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:26:52.553732 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee74e3_0158_4254_a21b_d9ce3300fe8b.slice/crio-c778315fae8741bad3d09d4b05cdb2455cf5c9e6791c28a41174ff0e523a7138 WatchSource:0}: Error finding container c778315fae8741bad3d09d4b05cdb2455cf5c9e6791c28a41174ff0e523a7138: Status 404 returned error can't find the container with id c778315fae8741bad3d09d4b05cdb2455cf5c9e6791c28a41174ff0e523a7138 Oct 30 01:26:52.561973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.561947 5548 generic.go:356] "Generic (PLEG): container finished" podID="76cf0ba4-b711-4a4a-902c-33bf164a5693" containerID="beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335" exitCode=0 Oct 30 01:26:52.562091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.561999 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" event={"ID":"76cf0ba4-b711-4a4a-902c-33bf164a5693","Type":"ContainerDied","Data":"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335"} Oct 30 01:26:52.562091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.562016 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" Oct 30 01:26:52.562091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.562047 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss" event={"ID":"76cf0ba4-b711-4a4a-902c-33bf164a5693","Type":"ContainerDied","Data":"268992659d383976835dc27ad39eab9d9eb10c647779703cf65310687cbadcf3"} Oct 30 01:26:52.562091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.562079 5548 scope.go:122] "RemoveContainer" containerID="beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335" Oct 30 01:26:52.585635 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.585610 5548 scope.go:122] "RemoveContainer" containerID="beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335" Oct 30 01:26:52.585965 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:26:52.585945 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335\": container with ID starting with beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335 not found: ID does not exist" containerID="beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335" Oct 30 01:26:52.586025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.585975 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335"} err="failed to get container status \"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335\": rpc error: code = NotFound desc = could not find container \"beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335\": container with ID starting with beb3fd5eb011a851c0a38b27ab9a9e28b8b6b2f16d8fa05068b9c1f892f13335 not found: ID does not exist" Oct 30 01:26:52.616540 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.616512 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:26:52.625552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:52.625529 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-wb4ss"] Oct 30 01:26:53.268344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:53.268313 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cf0ba4-b711-4a4a-902c-33bf164a5693" path="/var/lib/kubelet/pods/76cf0ba4-b711-4a4a-902c-33bf164a5693/volumes" Oct 30 01:26:53.570709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:53.570607 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" event={"ID":"f1ee74e3-0158-4254-a21b-d9ce3300fe8b","Type":"ContainerStarted","Data":"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95"} Oct 30 01:26:53.570709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:53.570667 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" event={"ID":"f1ee74e3-0158-4254-a21b-d9ce3300fe8b","Type":"ContainerStarted","Data":"c778315fae8741bad3d09d4b05cdb2455cf5c9e6791c28a41174ff0e523a7138"} Oct 30 01:26:53.570709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:53.570694 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:26:53.607668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:53.607620 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" podStartSLOduration=123.294043477 podStartE2EDuration="2m3.607606708s" totalImagesPullingTime="313.563231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:53.606698288 +0000 UTC m=+2604.449258881" watchObservedRunningTime="2026-10-30 01:26:53.607606708 +0000 UTC m=+2604.450167354" Oct 30 01:26:55.429157 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:26:55.429123 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:26:59.585362 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:26:59.585335 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:27:06.215984 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.215950 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:27:06.216792 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.216758 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" podUID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" containerName="maas-api" containerID="cri-o://0dc12db12f738f50e99e0fe403bcb86718265a631e054e5549e851435ebf1e7b" gracePeriod=30 Oct 30 01:27:06.730447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.730408 5548 generic.go:356] "Generic (PLEG): container finished" podID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" containerID="0dc12db12f738f50e99e0fe403bcb86718265a631e054e5549e851435ebf1e7b" exitCode=0 Oct 30 01:27:06.730999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.730733 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" event={"ID":"8d68a313-eb7f-4500-9f27-ec1337fcdb7b","Type":"ContainerDied","Data":"0dc12db12f738f50e99e0fe403bcb86718265a631e054e5549e851435ebf1e7b"} Oct 30 01:27:06.851207 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.851183 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:27:06.857696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.857671 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\" (UniqueName: \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\") pod \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " Oct 30 01:27:06.857847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.857727 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\") pod \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " Oct 30 01:27:06.857847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.857773 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") pod \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " Oct 30 01:27:06.858065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.858047 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\") pod \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\" (UID: \"8d68a313-eb7f-4500-9f27-ec1337fcdb7b\") " Oct 30 01:27:06.858229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.858197 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca" pod "8d68a313-eb7f-4500-9f27-ec1337fcdb7b" (UID: "8d68a313-eb7f-4500-9f27-ec1337fcdb7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:06.859055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.859029 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.860633 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.860608 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5" pod "8d68a313-eb7f-4500-9f27-ec1337fcdb7b" (UID: "8d68a313-eb7f-4500-9f27-ec1337fcdb7b"). InnerVolumeSpecName "kube-api-access-zflh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:06.861405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.861385 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs" pod "8d68a313-eb7f-4500-9f27-ec1337fcdb7b" (UID: "8d68a313-eb7f-4500-9f27-ec1337fcdb7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:06.861951 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.861928 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls" pod "8d68a313-eb7f-4500-9f27-ec1337fcdb7b" (UID: "8d68a313-eb7f-4500-9f27-ec1337fcdb7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:06.921229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.904432 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:27:06.921229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.905569 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" containerName="maas-api" Oct 30 01:27:06.921229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.905592 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" containerName="maas-api" Oct 30 01:27:06.921229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.905916 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" containerName="maas-api" Oct 30 01:27:06.921229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.911940 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:06.943641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.927211 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-73831d-serving-cert\"" Oct 30 01:27:06.943641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.943493 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:27:06.960592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960558 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:06.960828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960620 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5snj\" (UniqueName: \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:06.960828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960725 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:06.960828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960764 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:06.961049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960851 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.961049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960888 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zflh5\" (UniqueName: \"kubernetes.io/projected/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-kube-api-access-zflh5\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:06.961049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:06.960903 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d68a313-eb7f-4500-9f27-ec1337fcdb7b-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:07.062131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.062035 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5snj\" (UniqueName: \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.062326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.062188 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.062326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.062239 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.062534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.062478 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.063081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.063048 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.065737 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.065705 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.083502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.079968 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.091153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.091120 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5snj\" (UniqueName: \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\") pod \"maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.244419 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.244025 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:07.561344 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.561320 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:27:07.563086 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:27:07.563023 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba6f1aa_fc5b_420e_a649_94657c8f6a19.slice/crio-3e1ac57f82735e6f4f499e10f39f437488c8dd9263abde72eed2824558b93be0 WatchSource:0}: Error finding container 3e1ac57f82735e6f4f499e10f39f437488c8dd9263abde72eed2824558b93be0: Status 404 returned error can't find the container with id 3e1ac57f82735e6f4f499e10f39f437488c8dd9263abde72eed2824558b93be0 Oct 30 01:27:07.740107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.740072 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" Oct 30 01:27:07.740107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.740097 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr" event={"ID":"8d68a313-eb7f-4500-9f27-ec1337fcdb7b","Type":"ContainerDied","Data":"6a814c3cda2f11929970b3209733092ccb0c154eda8e84536ce0e69b3a251bef"} Oct 30 01:27:07.740386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.740158 5548 scope.go:122] "RemoveContainer" containerID="0dc12db12f738f50e99e0fe403bcb86718265a631e054e5549e851435ebf1e7b" Oct 30 01:27:07.741618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.741591 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" event={"ID":"5ba6f1aa-fc5b-420e-a649-94657c8f6a19","Type":"ContainerStarted","Data":"3e1ac57f82735e6f4f499e10f39f437488c8dd9263abde72eed2824558b93be0"} Oct 30 01:27:07.786562 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.786514 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:27:07.790407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:07.790378 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-eacea3-7f666dd955-xhgxr"] Oct 30 01:27:08.775394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:08.775341 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" event={"ID":"5ba6f1aa-fc5b-420e-a649-94657c8f6a19","Type":"ContainerStarted","Data":"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55"} Oct 30 01:27:08.775925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:08.775702 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:08.811156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:08.811101 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" podStartSLOduration=183.26856208 podStartE2EDuration="3m3.811083938s" totalImagesPullingTime="542.521858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:08.809479291 +0000 UTC m=+2619.652039884" watchObservedRunningTime="2026-10-30 01:27:08.811083938 +0000 UTC m=+2619.653644530" Oct 30 01:27:09.268096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:09.268067 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d68a313-eb7f-4500-9f27-ec1337fcdb7b" path="/var/lib/kubelet/pods/8d68a313-eb7f-4500-9f27-ec1337fcdb7b/volumes" Oct 30 01:27:11.069824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:11.069795 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:27:14.126342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:14.125221 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:27:14.126342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:14.125503 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" containerID="cri-o://b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d" gracePeriod=30 Oct 30 01:27:14.788910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:14.788878 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:27:16.070499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.070453 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" podUID="4f01b7d5-f187-4123-9b48-3698e7f94470" containerName="payload-processing" containerID="cri-o://e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b" gracePeriod=25 Oct 30 01:27:16.576387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.576357 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:27:16.722711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.722623 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\") pod \"4f01b7d5-f187-4123-9b48-3698e7f94470\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " Oct 30 01:27:16.722711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.722675 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\" (UniqueName: \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\") pod \"4f01b7d5-f187-4123-9b48-3698e7f94470\" (UID: \"4f01b7d5-f187-4123-9b48-3698e7f94470\") " Oct 30 01:27:16.723065 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.723039 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume" pod "4f01b7d5-f187-4123-9b48-3698e7f94470" (UID: "4f01b7d5-f187-4123-9b48-3698e7f94470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:16.723438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.723415 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f01b7d5-f187-4123-9b48-3698e7f94470-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:16.725142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.725113 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd" pod "4f01b7d5-f187-4123-9b48-3698e7f94470" (UID: "4f01b7d5-f187-4123-9b48-3698e7f94470"). InnerVolumeSpecName "kube-api-access-sglmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:16.824821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.824789 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sglmd\" (UniqueName: \"kubernetes.io/projected/4f01b7d5-f187-4123-9b48-3698e7f94470-kube-api-access-sglmd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:16.918194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.918162 5548 generic.go:356] "Generic (PLEG): container finished" podID="4f01b7d5-f187-4123-9b48-3698e7f94470" containerID="e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b" exitCode=0 Oct 30 01:27:16.918390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.918223 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" Oct 30 01:27:16.918390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.918255 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" event={"ID":"4f01b7d5-f187-4123-9b48-3698e7f94470","Type":"ContainerDied","Data":"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b"} Oct 30 01:27:16.918390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.918311 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb" event={"ID":"4f01b7d5-f187-4123-9b48-3698e7f94470","Type":"ContainerDied","Data":"ce3275f1efbe46f51414f62f5862b0ed8943a28f379591c899fb6062ba028252"} Oct 30 01:27:16.918390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.918334 5548 scope.go:122] "RemoveContainer" containerID="e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b" Oct 30 01:27:16.930072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.930055 5548 scope.go:122] "RemoveContainer" containerID="e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b" Oct 30 01:27:16.930354 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:16.930334 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b\": container with ID starting with e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b not found: ID does not exist" containerID="e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b" Oct 30 01:27:16.930448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.930368 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b"} err="failed to get container status \"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b\": rpc error: code = NotFound desc = could not find container \"e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b\": container with ID starting with e45d4d760beab7ad21f6e5e7001a06de36465a81d0a50a6344cffd126ef3568b not found: ID does not exist" Oct 30 01:27:16.964270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.964242 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:27:16.974364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:16.974296 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b650-8dd9774d8-shwkb"] Oct 30 01:27:17.266494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:17.266420 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f01b7d5-f187-4123-9b48-3698e7f94470" path="/var/lib/kubelet/pods/4f01b7d5-f187-4123-9b48-3698e7f94470/volumes" Oct 30 01:27:19.447741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.447702 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:27:19.469750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.469718 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.469932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.469769 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.469932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.469842 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470015 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470158 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470140 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470220 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470145 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:19.470220 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470196 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470336 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470254 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:19.470336 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470261 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470303 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:19.470445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470350 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470389 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:19.470445 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470416 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\") pod \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\" (UID: \"e4df4f73-9097-424d-a793-ccd0c1fb6b48\") " Oct 30 01:27:19.470655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.470452 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:19.471075 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.471053 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.471172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.471080 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.471172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.471094 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.471172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.471106 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.471172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.471118 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.472432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.472409 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:19.472825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.472806 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:19.473044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.473026 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:19.473098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.473070 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql" pod "e4df4f73-9097-424d-a793-ccd0c1fb6b48" (UID: "e4df4f73-9097-424d-a793-ccd0c1fb6b48"). InnerVolumeSpecName "kube-api-access-hwzql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:19.517271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.517231 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:27:19.518226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518190 5548 cpu_manager.go:414] "containerMap: removing container" podUID="4f01b7d5-f187-4123-9b48-3698e7f94470" containerName="payload-processing" Oct 30 01:27:19.518226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518217 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f01b7d5-f187-4123-9b48-3698e7f94470" containerName="payload-processing" Oct 30 01:27:19.518472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518238 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" Oct 30 01:27:19.518472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518244 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" Oct 30 01:27:19.518472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518434 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerName="istio-proxy" Oct 30 01:27:19.518472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.518464 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f01b7d5-f187-4123-9b48-3698e7f94470" containerName="payload-processing" Oct 30 01:27:19.523175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.523155 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.536162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.536136 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eaf753-openshift-default-dockercfg-jctct\"" Oct 30 01:27:19.539740 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.539716 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:27:19.572296 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572266 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572470 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572389 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572502 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572614 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572654 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzxh\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572770 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572849 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572800 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.572988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.572965 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.573052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.573024 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.573209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.573193 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.573279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.573216 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwzql\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-kube-api-access-hwzql\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.573279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.573230 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.573279 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.573244 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4df4f73-9097-424d-a793-ccd0c1fb6b48-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:19.674896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.674801 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.674896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.674840 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.674896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.674878 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzxh\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675223 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675013 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675223 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675042 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675223 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675177 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675224 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675306 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675353 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675394 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675473 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675607 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.675710 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675662 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.676005 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.675987 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.678008 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.677991 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.678221 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.678204 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.691923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.691896 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.692308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.692287 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzxh\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\") pod \"e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.847085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.847055 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:19.950880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.950794 5548 generic.go:356] "Generic (PLEG): container finished" podID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" containerID="b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d" exitCode=0 Oct 30 01:27:19.951030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.950890 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" Oct 30 01:27:19.951030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.951020 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" event={"ID":"e4df4f73-9097-424d-a793-ccd0c1fb6b48","Type":"ContainerDied","Data":"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d"} Oct 30 01:27:19.951112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.951073 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88" event={"ID":"e4df4f73-9097-424d-a793-ccd0c1fb6b48","Type":"ContainerDied","Data":"d05e38a669f8eceef4df649e2ba897bbd2415b38d880fcf8e9ddc0945e0c67b6"} Oct 30 01:27:19.951112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.951105 5548 scope.go:122] "RemoveContainer" containerID="b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d" Oct 30 01:27:19.968047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.968027 5548 scope.go:122] "RemoveContainer" containerID="b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d" Oct 30 01:27:19.968544 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:19.968524 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d\": container with ID starting with b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d not found: ID does not exist" containerID="b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d" Oct 30 01:27:19.968619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:19.968554 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d"} err="failed to get container status \"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d\": rpc error: code = NotFound desc = could not find container \"b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d\": container with ID starting with b9b58a007b4752e9103e8f4b58f9b4458db2cf975810847581063852d2d40e2d not found: ID does not exist" Oct 30 01:27:20.000700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.000671 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:27:20.009056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.009031 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-fpj88"] Oct 30 01:27:20.171873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.171829 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:27:20.173011 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:27:20.172980 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6a393e_6d18_4ee2_9c12_bf92f5d7e68e.slice/crio-99c33470d155ae3c784c558da9580ac1f765ca444b34d36881a73a0c20a2acbb WatchSource:0}: Error finding container 99c33470d155ae3c784c558da9580ac1f765ca444b34d36881a73a0c20a2acbb: Status 404 returned error can't find the container with id 99c33470d155ae3c784c558da9580ac1f765ca444b34d36881a73a0c20a2acbb Oct 30 01:27:20.176006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.175974 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:20.176096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.176036 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:20.176096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.176067 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:20.961457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.961418 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerStarted","Data":"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124"} Oct 30 01:27:20.961977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:20.961474 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerStarted","Data":"99c33470d155ae3c784c558da9580ac1f765ca444b34d36881a73a0c20a2acbb"} Oct 30 01:27:21.003509 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.003463 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podStartSLOduration=7.003448334 podStartE2EDuration="7.003448334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:20.998157727 +0000 UTC m=+2631.840718319" watchObservedRunningTime="2026-10-30 01:27:21.003448334 +0000 UTC m=+2631.846008927" Oct 30 01:27:21.268639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.268546 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4df4f73-9097-424d-a793-ccd0c1fb6b48" path="/var/lib/kubelet/pods/e4df4f73-9097-424d-a793-ccd0c1fb6b48/volumes" Oct 30 01:27:21.847663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.847629 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:21.854189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.854164 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:21.970762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.970722 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:21.971586 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:21.971568 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:27:24.231652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.231614 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:27:24.232195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.231852 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" podUID="bc37bf4c-08b6-4013-a029-58429ad97f72" containerName="maas-api" containerID="cri-o://d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0" gracePeriod=30 Oct 30 01:27:24.440164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.438491 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:27:24.440164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.438737 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" podUID="3bdec348-3d68-454d-9c07-18a6e0d90a49" containerName="payload-processing" containerID="cri-o://4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153" gracePeriod=30 Oct 30 01:27:24.584964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.584939 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:27:24.651153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.651109 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:27:24.652424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.652393 5548 cpu_manager.go:414] "containerMap: removing container" podUID="bc37bf4c-08b6-4013-a029-58429ad97f72" containerName="maas-api" Oct 30 01:27:24.652583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.652568 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc37bf4c-08b6-4013-a029-58429ad97f72" containerName="maas-api" Oct 30 01:27:24.652981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.652963 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc37bf4c-08b6-4013-a029-58429ad97f72" containerName="maas-api" Oct 30 01:27:24.657805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.657226 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.681578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.681557 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:27:24.681759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.681584 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\") pod \"bc37bf4c-08b6-4013-a029-58429ad97f72\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " Oct 30 01:27:24.681974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.681962 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\") pod \"bc37bf4c-08b6-4013-a029-58429ad97f72\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " Oct 30 01:27:24.682349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.682334 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\") pod \"bc37bf4c-08b6-4013-a029-58429ad97f72\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " Oct 30 01:27:24.682462 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.682451 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\") pod \"bc37bf4c-08b6-4013-a029-58429ad97f72\" (UID: \"bc37bf4c-08b6-4013-a029-58429ad97f72\") " Oct 30 01:27:24.683252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.682935 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca" pod "bc37bf4c-08b6-4013-a029-58429ad97f72" (UID: "bc37bf4c-08b6-4013-a029-58429ad97f72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:24.690001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.689983 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs" pod "bc37bf4c-08b6-4013-a029-58429ad97f72" (UID: "bc37bf4c-08b6-4013-a029-58429ad97f72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:24.690399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.690382 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2" pod "bc37bf4c-08b6-4013-a029-58429ad97f72" (UID: "bc37bf4c-08b6-4013-a029-58429ad97f72"). InnerVolumeSpecName "kube-api-access-smnb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:24.690580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.690560 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls" pod "bc37bf4c-08b6-4013-a029-58429ad97f72" (UID: "bc37bf4c-08b6-4013-a029-58429ad97f72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:24.830938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.800807 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850006 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850141 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850182 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850319 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850403 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850447 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850472 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqw7q\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850553 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850663 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/bc37bf4c-08b6-4013-a029-58429ad97f72-kube-api-access-smnb2\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850679 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc37bf4c-08b6-4013-a029-58429ad97f72-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850691 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:24.856045 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.850704 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc37bf4c-08b6-4013-a029-58429ad97f72-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:24.940966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.938050 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:27:24.951254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951227 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\" (UniqueName: \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\") pod \"3bdec348-3d68-454d-9c07-18a6e0d90a49\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " Oct 30 01:27:24.951429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951337 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\") pod \"3bdec348-3d68-454d-9c07-18a6e0d90a49\" (UID: \"3bdec348-3d68-454d-9c07-18a6e0d90a49\") " Oct 30 01:27:24.951707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951691 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.951796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951778 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.951902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951820 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume" pod "3bdec348-3d68-454d-9c07-18a6e0d90a49" (UID: "3bdec348-3d68-454d-9c07-18a6e0d90a49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:24.951987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.951965 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.952046 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.952018 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.952449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.952427 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953258 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953354 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953396 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953424 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqw7q\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953494 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.953665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953606 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdec348-3d68-454d-9c07-18a6e0d90a49-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:24.955743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.953907 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.955743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.954234 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.955743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.954755 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.955743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.954854 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.956192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.956175 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx" pod "3bdec348-3d68-454d-9c07-18a6e0d90a49" (UID: "3bdec348-3d68-454d-9c07-18a6e0d90a49"). InnerVolumeSpecName "kube-api-access-wp9gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:24.956310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.956288 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.956609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.956590 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.968919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.968898 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.971203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.971186 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqw7q\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.984119 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.984096 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:24.992809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.992775 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:27:24.993852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.993834 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3bdec348-3d68-454d-9c07-18a6e0d90a49" containerName="payload-processing" Oct 30 01:27:24.993852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.993853 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bdec348-3d68-454d-9c07-18a6e0d90a49" containerName="payload-processing" Oct 30 01:27:24.994053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:24.994036 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bdec348-3d68-454d-9c07-18a6e0d90a49" containerName="payload-processing" Oct 30 01:27:25.003100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.003077 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.007058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.006621 5548 generic.go:356] "Generic (PLEG): container finished" podID="bc37bf4c-08b6-4013-a029-58429ad97f72" containerID="d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0" exitCode=0 Oct 30 01:27:25.007058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.006686 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" Oct 30 01:27:25.007058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.006830 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" event={"ID":"bc37bf4c-08b6-4013-a029-58429ad97f72","Type":"ContainerDied","Data":"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0"} Oct 30 01:27:25.007058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.006919 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst" event={"ID":"bc37bf4c-08b6-4013-a029-58429ad97f72","Type":"ContainerDied","Data":"607b574376c0d6285a2562ea50b4f995dbc035ae70f91060396d003e4a071f82"} Oct 30 01:27:25.007058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.006948 5548 scope.go:122] "RemoveContainer" containerID="d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0" Oct 30 01:27:25.009297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.009264 5548 generic.go:356] "Generic (PLEG): container finished" podID="3bdec348-3d68-454d-9c07-18a6e0d90a49" containerID="4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153" exitCode=0 Oct 30 01:27:25.009405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.009350 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" event={"ID":"3bdec348-3d68-454d-9c07-18a6e0d90a49","Type":"ContainerDied","Data":"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153"} Oct 30 01:27:25.009405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.009394 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" event={"ID":"3bdec348-3d68-454d-9c07-18a6e0d90a49","Type":"ContainerDied","Data":"bff35155675c1adb4aa525956367a38e71317229a2feaf0a75c092950766b32f"} Oct 30 01:27:25.009532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.009466 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj" Oct 30 01:27:25.056447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.056399 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:27:25.057948 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.057915 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4rr\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.057989 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058156 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058123 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058163 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058201 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058317 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058281 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058315 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058388 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058446 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.058719 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.058694 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9gx\" (UniqueName: \"kubernetes.io/projected/3bdec348-3d68-454d-9c07-18a6e0d90a49-kube-api-access-wp9gx\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:25.079950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.079922 5548 scope.go:122] "RemoveContainer" containerID="d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0" Oct 30 01:27:25.080552 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:25.080519 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0\": container with ID starting with d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0 not found: ID does not exist" containerID="d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0" Oct 30 01:27:25.080735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.080705 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0"} err="failed to get container status \"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0\": rpc error: code = NotFound desc = could not find container \"d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0\": container with ID starting with d357a00b41652f60d2bb5fb4a38ce2a33b896e6ff1b827e31691dbcbf5aa71b0 not found: ID does not exist" Oct 30 01:27:25.080875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.080845 5548 scope.go:122] "RemoveContainer" containerID="4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153" Oct 30 01:27:25.099787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.099634 5548 scope.go:122] "RemoveContainer" containerID="4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153" Oct 30 01:27:25.100390 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:25.100348 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153\": container with ID starting with 4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153 not found: ID does not exist" containerID="4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153" Oct 30 01:27:25.100501 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.100401 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153"} err="failed to get container status \"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153\": rpc error: code = NotFound desc = could not find container \"4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153\": container with ID starting with 4e79b1e4c6f5826100ae740253ef1fc9c9776d16fa2ed63fbd03bd36b26ee153 not found: ID does not exist" Oct 30 01:27:25.117315 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.117264 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:27:25.126683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.126650 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4be164-788d9b6bcf-xnbst"] Oct 30 01:27:25.157431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.157372 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:27:25.160784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.159689 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.160784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.159734 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.160784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.159767 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.160784 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.159840 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.162211 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.162115 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.163130 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.162558 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.163327 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.163309 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.163490 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.163474 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.164492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.164475 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.165001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.164341 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.165001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.163830 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.165001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.164829 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.165354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.165335 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4rr\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.166054 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.165930 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.166430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.166392 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.166558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.166542 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.174763 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.174730 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be164-646f4df6bb-8bxsj"] Oct 30 01:27:25.182120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.182068 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.184400 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.184377 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4rr\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\") pod \"e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.270468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.270434 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bdec348-3d68-454d-9c07-18a6e0d90a49" path="/var/lib/kubelet/pods/3bdec348-3d68-454d-9c07-18a6e0d90a49/volumes" Oct 30 01:27:25.271110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.271090 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc37bf4c-08b6-4013-a029-58429ad97f72" path="/var/lib/kubelet/pods/bc37bf4c-08b6-4013-a029-58429ad97f72/volumes" Oct 30 01:27:25.353467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.353351 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:25.390020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.389989 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:27:25.397984 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.397934 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:25.398097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.398034 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:25.398097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.398077 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:25.440241 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:25.440204 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:27:25.698659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.698627 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:27:25.706308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.706269 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:25.706433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.706348 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:25.706433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:25.706380 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:27:26.062796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.062699 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" event={"ID":"5109d3d7-65d8-4e32-935b-b1813d768a65","Type":"ContainerStarted","Data":"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3"} Oct 30 01:27:26.062796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.062772 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" event={"ID":"5109d3d7-65d8-4e32-935b-b1813d768a65","Type":"ContainerStarted","Data":"67e287deca0556554edf1300a8f2d84ad728f48cd61d72d8984597827cc4414c"} Oct 30 01:27:26.069662 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.069622 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" event={"ID":"c379f8c4-ecb3-4241-8867-25faf70250a2","Type":"ContainerStarted","Data":"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821"} Oct 30 01:27:26.069830 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.069691 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" event={"ID":"c379f8c4-ecb3-4241-8867-25faf70250a2","Type":"ContainerStarted","Data":"8602d6b08fa60f8e1b81496a783d81ac64de0e175e0f8fe658766d781a75880b"} Oct 30 01:27:26.115636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.115568 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podStartSLOduration=219.115548623 podStartE2EDuration="3m39.115548623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:26.110337209 +0000 UTC m=+2636.952897802" watchObservedRunningTime="2026-10-30 01:27:26.115548623 +0000 UTC m=+2636.958109220" Oct 30 01:27:26.153118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.153072 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podStartSLOduration=209.153058828 podStartE2EDuration="3m29.153058828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:26.1497114 +0000 UTC m=+2636.992271992" watchObservedRunningTime="2026-10-30 01:27:26.153058828 +0000 UTC m=+2636.995619500" Oct 30 01:27:26.353095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.353012 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:26.354449 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.354418 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:26.354572 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.354485 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:26.987429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.987343 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:26.988086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.987712 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:26.988086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:26.987763 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:27.354793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:27.354709 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:27.354793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:27.354758 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:28.027883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.027833 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:28.028316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.028281 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:28.354918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.354771 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:28.354918 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.354835 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:28.999057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.999018 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:28.999356 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:28.999327 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:29.352873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:29.352777 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:29.352873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:29.352836 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:30.018940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.011002 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:30.018940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.011065 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:30.353134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.353058 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:30.353359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.353327 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:30.985887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.985277 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:30.985887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:30.985342 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:31.353398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:31.353306 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:31.354003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:31.353970 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:32.010583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.009379 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:27:32.010583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.010140 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:27:32.353841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.353741 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:27:32.353841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.353818 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 01:27:32.989649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.989614 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:32.990303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.990278 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:32.990933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:32.990918 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:27:33.360012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:33.359933 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:33.360494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:33.360477 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:33.361411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:33.361390 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:27:47.843903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:47.843818 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:27:47.844457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:47.844076 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" podUID="00325850-ceb0-418c-b0e6-f0b2332711b6" containerName="maas-api" containerID="cri-o://2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4" gracePeriod=30 Oct 30 01:27:48.424653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.424620 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:27:48.475507 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.475458 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:27:48.477179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.477148 5548 cpu_manager.go:414] "containerMap: removing container" podUID="00325850-ceb0-418c-b0e6-f0b2332711b6" containerName="maas-api" Oct 30 01:27:48.477179 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.477177 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00325850-ceb0-418c-b0e6-f0b2332711b6" containerName="maas-api" Oct 30 01:27:48.477399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.477363 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="00325850-ceb0-418c-b0e6-f0b2332711b6" containerName="maas-api" Oct 30 01:27:48.482168 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.482142 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.487084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.487003 5548 generic.go:356] "Generic (PLEG): container finished" podID="00325850-ceb0-418c-b0e6-f0b2332711b6" containerID="2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4" exitCode=0 Oct 30 01:27:48.487235 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.487149 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" Oct 30 01:27:48.487235 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.487200 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" event={"ID":"00325850-ceb0-418c-b0e6-f0b2332711b6","Type":"ContainerDied","Data":"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4"} Oct 30 01:27:48.487369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.487251 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl" event={"ID":"00325850-ceb0-418c-b0e6-f0b2332711b6","Type":"ContainerDied","Data":"f16090bf53c39f74c19d33b5997b6aed5843156351d477ed44632f0b6bd80bb3"} Oct 30 01:27:48.487369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.487282 5548 scope.go:122] "RemoveContainer" containerID="2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4" Oct 30 01:27:48.496451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.495641 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-29bb56-serving-cert\"" Oct 30 01:27:48.503610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.503320 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.581910 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\") pod \"00325850-ceb0-418c-b0e6-f0b2332711b6\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582022 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\") pod \"00325850-ceb0-418c-b0e6-f0b2332711b6\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582069 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\") pod \"00325850-ceb0-418c-b0e6-f0b2332711b6\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582195 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\" (UniqueName: \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\") pod \"00325850-ceb0-418c-b0e6-f0b2332711b6\" (UID: \"00325850-ceb0-418c-b0e6-f0b2332711b6\") " Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582635 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582817 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.582849 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwh7\" (UniqueName: \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.587892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.583049 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.614258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.595393 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca" pod "00325850-ceb0-418c-b0e6-f0b2332711b6" (UID: "00325850-ceb0-418c-b0e6-f0b2332711b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:48.627547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.616893 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls" pod "00325850-ceb0-418c-b0e6-f0b2332711b6" (UID: "00325850-ceb0-418c-b0e6-f0b2332711b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:48.627547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.618771 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs" pod "00325850-ceb0-418c-b0e6-f0b2332711b6" (UID: "00325850-ceb0-418c-b0e6-f0b2332711b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:48.627547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.624057 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj" pod "00325850-ceb0-418c-b0e6-f0b2332711b6" (UID: "00325850-ceb0-418c-b0e6-f0b2332711b6"). InnerVolumeSpecName "kube-api-access-jd9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:48.670777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.668514 5548 scope.go:122] "RemoveContainer" containerID="2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4" Oct 30 01:27:48.670777 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:48.670639 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4\": container with ID starting with 2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4 not found: ID does not exist" containerID="2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4" Oct 30 01:27:48.670777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.670688 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4"} err="failed to get container status \"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4\": rpc error: code = NotFound desc = could not find container \"2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4\": container with ID starting with 2bd289feead3c7f3a4c621a3f6ef33da09ba90695a26481e0f237e95a05916b4 not found: ID does not exist" Oct 30 01:27:48.685187 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685145 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.685380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685314 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.685491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685476 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.685561 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685512 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwh7\" (UniqueName: \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.685623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685592 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.685623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685608 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00325850-ceb0-418c-b0e6-f0b2332711b6-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.685735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685621 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00325850-ceb0-418c-b0e6-f0b2332711b6-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.685735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.685635 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd9rj\" (UniqueName: \"kubernetes.io/projected/00325850-ceb0-418c-b0e6-f0b2332711b6-kube-api-access-jd9rj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:48.686120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.686093 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.688350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.688327 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.688912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.688839 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.703904 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.703836 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwh7\" (UniqueName: \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\") pod \"maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.813644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.813609 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:48.832354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.832318 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:27:48.836575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:48.836544 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-69dc90-76d8546585-mhrkl"] Oct 30 01:27:49.132732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:49.132706 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:27:49.133203 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:27:49.133094 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a5a2ec_5a4a_48a9_8841_8123894ba61f.slice/crio-42fca97c527552611c7757ab44929b2b323cce23b9eb8927eccbdcadcc38a2cc WatchSource:0}: Error finding container 42fca97c527552611c7757ab44929b2b323cce23b9eb8927eccbdcadcc38a2cc: Status 404 returned error can't find the container with id 42fca97c527552611c7757ab44929b2b323cce23b9eb8927eccbdcadcc38a2cc Oct 30 01:27:49.270055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:49.269955 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00325850-ceb0-418c-b0e6-f0b2332711b6" path="/var/lib/kubelet/pods/00325850-ceb0-418c-b0e6-f0b2332711b6/volumes" Oct 30 01:27:49.506708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:49.506673 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" event={"ID":"89a5a2ec-5a4a-48a9-8841-8123894ba61f","Type":"ContainerStarted","Data":"42fca97c527552611c7757ab44929b2b323cce23b9eb8927eccbdcadcc38a2cc"} Oct 30 01:27:50.518037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:50.518001 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" event={"ID":"89a5a2ec-5a4a-48a9-8841-8123894ba61f","Type":"ContainerStarted","Data":"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20"} Oct 30 01:27:50.518499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:50.518063 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:50.556623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:50.556571 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" podStartSLOduration=21.241216123 podStartE2EDuration="21.556554315s" totalImagesPullingTime="315.338192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:50.554170454 +0000 UTC m=+2661.396731047" watchObservedRunningTime="2026-10-30 01:27:50.556554315 +0000 UTC m=+2661.399114930" Oct 30 01:27:55.657200 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:55.657155 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:27:56.530414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:56.530386 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:27:56.756657 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:56.756626 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:27:56.757206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:56.756886 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" podUID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" containerName="maas-api" containerID="cri-o://edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95" gracePeriod=30 Oct 30 01:27:57.338883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.335162 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:27:57.498396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.498288 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\") pod \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " Oct 30 01:27:57.498396 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.498393 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\") pod \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " Oct 30 01:27:57.498653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.498484 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\" (UniqueName: \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\") pod \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " Oct 30 01:27:57.498653 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.498527 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\") pod \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\" (UID: \"f1ee74e3-0158-4254-a21b-d9ce3300fe8b\") " Oct 30 01:27:57.499371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.499341 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca" pod "f1ee74e3-0158-4254-a21b-d9ce3300fe8b" (UID: "f1ee74e3-0158-4254-a21b-d9ce3300fe8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:57.502533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.502474 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls" pod "f1ee74e3-0158-4254-a21b-d9ce3300fe8b" (UID: "f1ee74e3-0158-4254-a21b-d9ce3300fe8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:57.502715 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.502560 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76" pod "f1ee74e3-0158-4254-a21b-d9ce3300fe8b" (UID: "f1ee74e3-0158-4254-a21b-d9ce3300fe8b"). InnerVolumeSpecName "kube-api-access-n4b76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:57.512020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.511988 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs" pod "f1ee74e3-0158-4254-a21b-d9ce3300fe8b" (UID: "f1ee74e3-0158-4254-a21b-d9ce3300fe8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:57.600999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.600966 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4b76\" (UniqueName: \"kubernetes.io/projected/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-kube-api-access-n4b76\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:57.601209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.601191 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:57.601297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.601214 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:57.601297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.601229 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee74e3-0158-4254-a21b-d9ce3300fe8b-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:27:57.611181 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.611153 5548 generic.go:356] "Generic (PLEG): container finished" podID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" containerID="edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95" exitCode=0 Oct 30 01:27:57.611360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.611205 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" Oct 30 01:27:57.611360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.611207 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" event={"ID":"f1ee74e3-0158-4254-a21b-d9ce3300fe8b","Type":"ContainerDied","Data":"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95"} Oct 30 01:27:57.611360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.611325 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx" event={"ID":"f1ee74e3-0158-4254-a21b-d9ce3300fe8b","Type":"ContainerDied","Data":"c778315fae8741bad3d09d4b05cdb2455cf5c9e6791c28a41174ff0e523a7138"} Oct 30 01:27:57.611360 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.611354 5548 scope.go:122] "RemoveContainer" containerID="edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95" Oct 30 01:27:57.669439 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.667722 5548 scope.go:122] "RemoveContainer" containerID="edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95" Oct 30 01:27:57.670533 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:27:57.670136 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95\": container with ID starting with edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95 not found: ID does not exist" containerID="edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95" Oct 30 01:27:57.670533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.670178 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95"} err="failed to get container status \"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95\": rpc error: code = NotFound desc = could not find container \"edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95\": container with ID starting with edccd47f85512a825b72c9a86846b0140596441c9fcfedb3ed329a2434fb3d95 not found: ID does not exist" Oct 30 01:27:57.686990 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.686952 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:27:57.693097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:57.693054 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b650-667c786b6c-pjdtx"] Oct 30 01:27:59.365925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.360927 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" path="/var/lib/kubelet/pods/f1ee74e3-0158-4254-a21b-d9ce3300fe8b/volumes" Oct 30 01:27:59.365925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.361416 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:27:59.365925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.362438 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" containerName="maas-api" Oct 30 01:27:59.365925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.362460 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" containerName="maas-api" Oct 30 01:27:59.365925 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.362639 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ee74e3-0158-4254-a21b-d9ce3300fe8b" containerName="maas-api" Oct 30 01:27:59.408894 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.400113 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:27:59.408894 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.400256 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.460883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.456402 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-368bbb-dockercfg-xfncd\"" Oct 30 01:27:59.503921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.503371 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-368bbb\"" Oct 30 01:27:59.511789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.511711 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:27:59.524284 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.524238 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:27:59.524482 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.524378 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.563722 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.563688 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwn8\" (UniqueName: \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.564222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.564179 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.665966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.665934 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:27:59.666161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.665988 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwn8\" (UniqueName: \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.666161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.666047 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.666294 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.666204 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.666355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.666319 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h665x\" (UniqueName: \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.667269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.667246 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.668844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.668774 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerName="maas-api" containerID="cri-o://e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55" gracePeriod=30 Oct 30 01:27:59.711771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.711736 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwn8\" (UniqueName: \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\") pod \"payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.767965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.767926 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h665x\" (UniqueName: \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.768130 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.768091 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.769044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.769015 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.783236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.783201 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.156:8443/health\": dial tcp 10.134.0.156:8443: connect: connection refused" Oct 30 01:27:59.785890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.785846 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h665x\" (UniqueName: \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\") pod \"payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:27:59.812995 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.812961 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:27:59.843259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:27:59.843059 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:28:00.137253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.137227 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:28:00.200549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.200507 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\") pod \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " Oct 30 01:28:00.200747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.200576 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\" (UniqueName: \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\") pod \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " Oct 30 01:28:00.200989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.200929 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\") pod \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " Oct 30 01:28:00.201101 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.200997 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\") pod \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\" (UID: \"5ba6f1aa-fc5b-420e-a649-94657c8f6a19\") " Oct 30 01:28:00.202036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.201879 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca" pod "5ba6f1aa-fc5b-420e-a649-94657c8f6a19" (UID: "5ba6f1aa-fc5b-420e-a649-94657c8f6a19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.203664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.203640 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj" pod "5ba6f1aa-fc5b-420e-a649-94657c8f6a19" (UID: "5ba6f1aa-fc5b-420e-a649-94657c8f6a19"). InnerVolumeSpecName "kube-api-access-s5snj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.204034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.204010 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs" pod "5ba6f1aa-fc5b-420e-a649-94657c8f6a19" (UID: "5ba6f1aa-fc5b-420e-a649-94657c8f6a19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:00.205174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.205153 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls" pod "5ba6f1aa-fc5b-420e-a649-94657c8f6a19" (UID: "5ba6f1aa-fc5b-420e-a649-94657c8f6a19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:00.303944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.303905 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.303944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.303948 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.304196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.303963 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.304196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.303977 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5snj\" (UniqueName: \"kubernetes.io/projected/5ba6f1aa-fc5b-420e-a649-94657c8f6a19-kube-api-access-s5snj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.351311 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:28:00.351262 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6504af3_1d50_4b0f_9e02_043033355293.slice/crio-2556a218d8fd9989add87aa48c55540df2fb56a5a90134e273a4d3db130a134e WatchSource:0}: Error finding container 2556a218d8fd9989add87aa48c55540df2fb56a5a90134e273a4d3db130a134e: Status 404 returned error can't find the container with id 2556a218d8fd9989add87aa48c55540df2fb56a5a90134e273a4d3db130a134e Oct 30 01:28:00.352086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.352059 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:28:00.384823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.384796 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:28:00.397301 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:28:00.397274 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083f1262_dc88_489a_b84d_f0d76c32ab4f.slice/crio-4adc0a216f479721f3b7069005b22e4bb778e04dca38c11fce6b13cd49218b19 WatchSource:0}: Error finding container 4adc0a216f479721f3b7069005b22e4bb778e04dca38c11fce6b13cd49218b19: Status 404 returned error can't find the container with id 4adc0a216f479721f3b7069005b22e4bb778e04dca38c11fce6b13cd49218b19 Oct 30 01:28:00.638917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.638822 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" event={"ID":"083f1262-dc88-489a-b84d-f0d76c32ab4f","Type":"ContainerStarted","Data":"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a"} Oct 30 01:28:00.638917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.638906 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:28:00.639136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.638925 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" event={"ID":"083f1262-dc88-489a-b84d-f0d76c32ab4f","Type":"ContainerStarted","Data":"4adc0a216f479721f3b7069005b22e4bb778e04dca38c11fce6b13cd49218b19"} Oct 30 01:28:00.640487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.640461 5548 generic.go:356] "Generic (PLEG): container finished" podID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerID="e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55" exitCode=0 Oct 30 01:28:00.640623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.640522 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" Oct 30 01:28:00.640623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.640546 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" event={"ID":"5ba6f1aa-fc5b-420e-a649-94657c8f6a19","Type":"ContainerDied","Data":"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55"} Oct 30 01:28:00.640623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.640602 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv" event={"ID":"5ba6f1aa-fc5b-420e-a649-94657c8f6a19","Type":"ContainerDied","Data":"3e1ac57f82735e6f4f499e10f39f437488c8dd9263abde72eed2824558b93be0"} Oct 30 01:28:00.640823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.640631 5548 scope.go:122] "RemoveContainer" containerID="e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55" Oct 30 01:28:00.642838 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.642811 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" event={"ID":"c6504af3-1d50-4b0f-9e02-043033355293","Type":"ContainerStarted","Data":"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947"} Oct 30 01:28:00.643062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.642921 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" event={"ID":"c6504af3-1d50-4b0f-9e02-043033355293","Type":"ContainerStarted","Data":"2556a218d8fd9989add87aa48c55540df2fb56a5a90134e273a4d3db130a134e"} Oct 30 01:28:00.643062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.642958 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:28:00.652453 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.652437 5548 scope.go:122] "RemoveContainer" containerID="e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55" Oct 30 01:28:00.652775 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:00.652752 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55\": container with ID starting with e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55 not found: ID does not exist" containerID="e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55" Oct 30 01:28:00.652840 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.652786 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55"} err="failed to get container status \"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55\": rpc error: code = NotFound desc = could not find container \"e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55\": container with ID starting with e132bcb7373de51b254bb4b6b0b56c94931c572d6c05508b57c1cb035e41ef55 not found: ID does not exist" Oct 30 01:28:00.684093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.684028 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" podStartSLOduration=1.684010259 podStartE2EDuration="1.684010259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:00.679953577 +0000 UTC m=+2671.522514170" watchObservedRunningTime="2026-10-30 01:28:00.684010259 +0000 UTC m=+2671.526570855" Oct 30 01:28:00.716956 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.716855 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" podStartSLOduration=1.716837704 podStartE2EDuration="1.716837704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:00.715994916 +0000 UTC m=+2671.558555509" watchObservedRunningTime="2026-10-30 01:28:00.716837704 +0000 UTC m=+2671.559398287" Oct 30 01:28:00.762022 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.761970 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:28:00.763832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.763803 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73831d-5f5b7978c5-c5tgv"] Oct 30 01:28:00.805827 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:00.805792 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 01:28:01.305885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:01.301538 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" path="/var/lib/kubelet/pods/5ba6f1aa-fc5b-420e-a649-94657c8f6a19/volumes" Oct 30 01:28:04.621131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:04.620893 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:28:04.622067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:04.621151 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" containerID="cri-o://540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a" gracePeriod=30 Oct 30 01:28:09.982778 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:09.982744 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:28:10.024828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.024801 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.024846 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.024925 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.024944 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.024963 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025066 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025133 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025172 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025257 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\") pod \"a16525c9-524a-4635-8057-9712f693e64f\" (UID: \"a16525c9-524a-4635-8057-9712f693e64f\") " Oct 30 01:28:10.025493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025443 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:10.025644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025617 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:10.025936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025640 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:10.025936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025815 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:10.025936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.025845 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:10.026436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.026411 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.026527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.026440 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.026527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.026455 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.026527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.026468 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16525c9-524a-4635-8057-9712f693e64f-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.026527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.026482 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.027428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.027405 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:10.027512 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.027458 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:10.027598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.027585 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:10.027732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.027718 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs" pod "a16525c9-524a-4635-8057-9712f693e64f" (UID: "a16525c9-524a-4635-8057-9712f693e64f"). InnerVolumeSpecName "kube-api-access-fwxgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:10.128614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.128583 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16525c9-524a-4635-8057-9712f693e64f-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.128811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.128631 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxgs\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-kube-api-access-fwxgs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.128811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.128642 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16525c9-524a-4635-8057-9712f693e64f-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.128811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.128650 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16525c9-524a-4635-8057-9712f693e64f-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:10.775399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.775357 5548 generic.go:356] "Generic (PLEG): container finished" podID="a16525c9-524a-4635-8057-9712f693e64f" containerID="540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a" exitCode=0 Oct 30 01:28:10.775590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.775406 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" Oct 30 01:28:10.775590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.775445 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerDied","Data":"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a"} Oct 30 01:28:10.775590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.775519 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v" event={"ID":"a16525c9-524a-4635-8057-9712f693e64f","Type":"ContainerDied","Data":"439a0f35bf1014700e3578265bab8fcc481f0749822623accd536a743c2bbeb6"} Oct 30 01:28:10.775590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.775554 5548 scope.go:122] "RemoveContainer" containerID="540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a" Oct 30 01:28:10.789003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.788984 5548 scope.go:122] "RemoveContainer" containerID="c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b" Oct 30 01:28:10.800453 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.800427 5548 scope.go:122] "RemoveContainer" containerID="540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a" Oct 30 01:28:10.800870 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:10.800816 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a\": container with ID starting with 540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a not found: ID does not exist" containerID="540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a" Oct 30 01:28:10.800987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.800915 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a"} err="failed to get container status \"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a\": rpc error: code = NotFound desc = could not find container \"540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a\": container with ID starting with 540e02d498755b3cdf212f620152f9fa858847ac80f7f196b6f0e0cb9af9c64a not found: ID does not exist" Oct 30 01:28:10.800987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.800944 5548 scope.go:122] "RemoveContainer" containerID="c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b" Oct 30 01:28:10.801290 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:10.801262 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b\": container with ID starting with c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b not found: ID does not exist" containerID="c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b" Oct 30 01:28:10.801369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.801302 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b"} err="failed to get container status \"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b\": rpc error: code = NotFound desc = could not find container \"c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b\": container with ID starting with c9755b97be7f51c43ff1da9f5b1e0a1b38ac757d878aadcb7640961dd9571b0b not found: ID does not exist" Oct 30 01:28:10.831937 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.831907 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:28:10.841263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:10.841242 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6b650-openshift-default-6bc5bcc7dc-z2d6v"] Oct 30 01:28:11.269393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:11.269358 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16525c9-524a-4635-8057-9712f693e64f" path="/var/lib/kubelet/pods/a16525c9-524a-4635-8057-9712f693e64f/volumes" Oct 30 01:28:11.655569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:11.655545 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:28:11.655729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:11.655610 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:28:16.005885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.000236 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" containerID="cri-o://efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4" gracePeriod=45 Oct 30 01:28:16.051760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.051068 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": read tcp 10.134.0.2:43382->10.134.0.45:8000: read: connection reset by peer" Oct 30 01:28:16.396608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.396583 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:28:16.502880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.502832 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.503154 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503131 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.503239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503199 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.503239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503217 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:16.503365 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503334 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\" (UniqueName: \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.503423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503409 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:16.503483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503428 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.503483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.503468 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\") pod \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\" (UID: \"5fd5cd32-44ce-4ab0-8120-49af64d8dc02\") " Oct 30 01:28:16.505516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.504952 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:16.505516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.504981 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:16.505516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.505172 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:16.523416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.517842 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:16.523416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.518436 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:16.523416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.519995 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8" pod "5fd5cd32-44ce-4ab0-8120-49af64d8dc02" (UID: "5fd5cd32-44ce-4ab0-8120-49af64d8dc02"). InnerVolumeSpecName "kube-api-access-b6bc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:16.618968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.611756 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:16.619226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.619202 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bc8\" (UniqueName: \"kubernetes.io/projected/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-kube-api-access-b6bc8\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:16.619226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.619225 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:16.619407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:16.619236 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fd5cd32-44ce-4ab0-8120-49af64d8dc02-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:17.020960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.020921 5548 generic.go:356] "Generic (PLEG): container finished" podID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerID="efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4" exitCode=0 Oct 30 01:28:17.021756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.020977 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" Oct 30 01:28:17.021756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.020978 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" event={"ID":"5fd5cd32-44ce-4ab0-8120-49af64d8dc02","Type":"ContainerDied","Data":"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4"} Oct 30 01:28:17.021756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.021034 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4" event={"ID":"5fd5cd32-44ce-4ab0-8120-49af64d8dc02","Type":"ContainerDied","Data":"d4d425f73f63a64890b15b50e90bada3f71dd02adf2b4c341a5f0f5144d6e36d"} Oct 30 01:28:17.021756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.021063 5548 scope.go:122] "RemoveContainer" containerID="efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4" Oct 30 01:28:17.035273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.035257 5548 scope.go:122] "RemoveContainer" containerID="efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4" Oct 30 01:28:17.035586 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:17.035566 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4\": container with ID starting with efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4 not found: ID does not exist" containerID="efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4" Oct 30 01:28:17.035679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.035594 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4"} err="failed to get container status \"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4\": rpc error: code = NotFound desc = could not find container \"efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4\": container with ID starting with efdbcb06129c64fdb4f1da8d3867c457a87dcb6e78b2e7b6159aa2b69f9552c4 not found: ID does not exist" Oct 30 01:28:17.076680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.076649 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 01:28:17.095446 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.095417 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ba3d7b/simulator-w0-ba3d7b-kserve-759864dbcd-9lvs4"] Oct 30 01:28:17.300815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:17.300738 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" path="/var/lib/kubelet/pods/5fd5cd32-44ce-4ab0-8120-49af64d8dc02/volumes" Oct 30 01:28:21.993757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:21.993722 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:28:22.005884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:21.994567 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" podUID="5b108e37-6075-4bb1-9204-bdf0712a9c05" containerName="maas-api" containerID="cri-o://edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa" gracePeriod=30 Oct 30 01:28:22.610885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.609763 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:28:22.703131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.703091 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\") pod \"5b108e37-6075-4bb1-9204-bdf0712a9c05\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " Oct 30 01:28:22.703337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.703196 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\") pod \"5b108e37-6075-4bb1-9204-bdf0712a9c05\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " Oct 30 01:28:22.703337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.703332 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") pod \"5b108e37-6075-4bb1-9204-bdf0712a9c05\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " Oct 30 01:28:22.703457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.703373 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\" (UniqueName: \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\") pod \"5b108e37-6075-4bb1-9204-bdf0712a9c05\" (UID: \"5b108e37-6075-4bb1-9204-bdf0712a9c05\") " Oct 30 01:28:22.704934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.704827 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca" pod "5b108e37-6075-4bb1-9204-bdf0712a9c05" (UID: "5b108e37-6075-4bb1-9204-bdf0712a9c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:22.712086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.712037 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs" pod "5b108e37-6075-4bb1-9204-bdf0712a9c05" (UID: "5b108e37-6075-4bb1-9204-bdf0712a9c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:22.715260 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.715233 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls" pod "5b108e37-6075-4bb1-9204-bdf0712a9c05" (UID: "5b108e37-6075-4bb1-9204-bdf0712a9c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:22.715999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.715977 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht" pod "5b108e37-6075-4bb1-9204-bdf0712a9c05" (UID: "5b108e37-6075-4bb1-9204-bdf0712a9c05"). InnerVolumeSpecName "kube-api-access-mnxht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:22.819441 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.806377 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:22.819719 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.819693 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxht\" (UniqueName: \"kubernetes.io/projected/5b108e37-6075-4bb1-9204-bdf0712a9c05-kube-api-access-mnxht\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:22.819826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.819813 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b108e37-6075-4bb1-9204-bdf0712a9c05-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:22.819944 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:22.819931 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b108e37-6075-4bb1-9204-bdf0712a9c05-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:23.103986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.103952 5548 generic.go:356] "Generic (PLEG): container finished" podID="5b108e37-6075-4bb1-9204-bdf0712a9c05" containerID="edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa" exitCode=0 Oct 30 01:28:23.104460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.104008 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" Oct 30 01:28:23.104460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.104046 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" event={"ID":"5b108e37-6075-4bb1-9204-bdf0712a9c05","Type":"ContainerDied","Data":"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa"} Oct 30 01:28:23.104460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.104139 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h" event={"ID":"5b108e37-6075-4bb1-9204-bdf0712a9c05","Type":"ContainerDied","Data":"934e1b0e6cff57482ad4ba2dba43a1f997cd8d95534254b53f3ea246aaf42a5b"} Oct 30 01:28:23.104460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.104167 5548 scope.go:122] "RemoveContainer" containerID="edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa" Oct 30 01:28:23.117415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.117389 5548 scope.go:122] "RemoveContainer" containerID="edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa" Oct 30 01:28:23.117724 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:23.117705 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa\": container with ID starting with edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa not found: ID does not exist" containerID="edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa" Oct 30 01:28:23.117774 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.117735 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa"} err="failed to get container status \"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa\": rpc error: code = NotFound desc = could not find container \"edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa\": container with ID starting with edbf0c995bb3c043215a68686c01d5a26a3e57f03e81d06ed497f8f78d0107fa not found: ID does not exist" Oct 30 01:28:23.157711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.157683 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:28:23.168356 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.168325 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-18a189-845794cf99-dkz9h"] Oct 30 01:28:23.275523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:23.275481 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b108e37-6075-4bb1-9204-bdf0712a9c05" path="/var/lib/kubelet/pods/5b108e37-6075-4bb1-9204-bdf0712a9c05/volumes" Oct 30 01:28:25.428518 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:25.428469 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:28:30.880280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:30.880241 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:28:30.886039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:30.886004 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:28:38.034844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:38.033151 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:28:38.034844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:38.033227 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:28:42.399493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.399449 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:42.399943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.399511 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:42.476883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.475983 5548 generic.go:356] "Generic (PLEG): container finished" podID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerID="50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a" exitCode=0 Oct 30 01:28:42.476883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.476085 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerDied","Data":"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a"} Oct 30 01:28:42.476883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.476656 5548 scope.go:122] "RemoveContainer" containerID="50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a" Oct 30 01:28:42.492906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.491088 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:42.492906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.491196 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:42.492906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.491250 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:42.568092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.560233 5548 generic.go:356] "Generic (PLEG): container finished" podID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerID="c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124" exitCode=0 Oct 30 01:28:42.580816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.580766 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerDied","Data":"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124"} Oct 30 01:28:42.581443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.581360 5548 scope.go:122] "RemoveContainer" containerID="c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124" Oct 30 01:28:42.597642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.597595 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:42.597796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.597695 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:42.597796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:42.597764 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:28:43.625576 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.625540 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerStarted","Data":"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366"} Oct 30 01:28:43.628061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.628036 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerStarted","Data":"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c"} Oct 30 01:28:43.747300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.747195 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:28:43.747618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.747593 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:43.747759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.747647 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:43.847899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.847853 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:28:43.848137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.848113 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:43.848228 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:43.848169 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:44.747298 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:44.747270 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:44.747741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:44.747712 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:44.848187 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:44.848100 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:44.848187 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:44.848157 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:45.756886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:45.753303 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:45.756886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:45.753364 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:45.847465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:45.847437 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:45.847691 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:45.847668 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:46.746988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:46.746951 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:46.747174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:46.747015 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:46.848189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:46.848152 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:46.848680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:46.848216 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:47.747310 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:47.747274 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:47.747487 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:47.747340 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:47.848254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:47.848216 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:47.848665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:47.848273 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:48.753465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:48.753428 5548 patch_prober.go:29] interesting pod/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 01:28:48.753741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:48.753715 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 01:28:48.847491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:48.847463 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:28:48.847687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:48.847667 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 01:28:49.752596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.752566 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:28:49.753007 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.752956 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:28:49.753574 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.753554 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:28:49.869200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.869173 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:28:49.869993 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.869964 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:28:49.871442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:49.871421 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:28:50.342378 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.342343 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:28:50.343232 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.343173 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" podUID="16f9868e-83be-4b11-a343-4b0609e143b5" containerName="maas-api" containerID="cri-o://6c89cdc310ed786327deb5cf1d6033ac651f5c165eb5171de1a8a8cafe409af0" gracePeriod=30 Oct 30 01:28:50.711303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.708456 5548 generic.go:356] "Generic (PLEG): container finished" podID="16f9868e-83be-4b11-a343-4b0609e143b5" containerID="6c89cdc310ed786327deb5cf1d6033ac651f5c165eb5171de1a8a8cafe409af0" exitCode=0 Oct 30 01:28:50.711303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.708537 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" event={"ID":"16f9868e-83be-4b11-a343-4b0609e143b5","Type":"ContainerDied","Data":"6c89cdc310ed786327deb5cf1d6033ac651f5c165eb5171de1a8a8cafe409af0"} Oct 30 01:28:50.742337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.742255 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:28:50.941436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.867209 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\") pod \"16f9868e-83be-4b11-a343-4b0609e143b5\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " Oct 30 01:28:50.941436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.867271 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\") pod \"16f9868e-83be-4b11-a343-4b0609e143b5\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " Oct 30 01:28:50.941436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.867330 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") pod \"16f9868e-83be-4b11-a343-4b0609e143b5\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " Oct 30 01:28:50.941436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.867407 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\" (UniqueName: \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\") pod \"16f9868e-83be-4b11-a343-4b0609e143b5\" (UID: \"16f9868e-83be-4b11-a343-4b0609e143b5\") " Oct 30 01:28:50.941436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.936263 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4" pod "16f9868e-83be-4b11-a343-4b0609e143b5" (UID: "16f9868e-83be-4b11-a343-4b0609e143b5"). InnerVolumeSpecName "kube-api-access-nc7w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:50.980895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:50.980602 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc7w4\" (UniqueName: \"kubernetes.io/projected/16f9868e-83be-4b11-a343-4b0609e143b5-kube-api-access-nc7w4\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.015262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.014219 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs" pod "16f9868e-83be-4b11-a343-4b0609e143b5" (UID: "16f9868e-83be-4b11-a343-4b0609e143b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:51.015644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.015605 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca" pod "16f9868e-83be-4b11-a343-4b0609e143b5" (UID: "16f9868e-83be-4b11-a343-4b0609e143b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:51.201934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.190090 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls" pod "16f9868e-83be-4b11-a343-4b0609e143b5" (UID: "16f9868e-83be-4b11-a343-4b0609e143b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:51.276892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.274307 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f9868e-83be-4b11-a343-4b0609e143b5-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.276892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.274782 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.276892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.275232 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16f9868e-83be-4b11-a343-4b0609e143b5-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.749692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.749655 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" event={"ID":"16f9868e-83be-4b11-a343-4b0609e143b5","Type":"ContainerDied","Data":"450ced9b33a9ddaf51e8a3a4807fec5158bb0cc030f5509198f47040c7f61cd7"} Oct 30 01:28:51.749886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.749722 5548 scope.go:122] "RemoveContainer" containerID="6c89cdc310ed786327deb5cf1d6033ac651f5c165eb5171de1a8a8cafe409af0" Oct 30 01:28:51.749886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.749851 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788" Oct 30 01:28:51.798772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.798722 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:28:51.804605 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:51.804566 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-262230-677578987f-nl788"] Oct 30 01:28:53.275614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:53.275575 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f9868e-83be-4b11-a343-4b0609e143b5" path="/var/lib/kubelet/pods/16f9868e-83be-4b11-a343-4b0609e143b5/volumes" Oct 30 01:28:55.432478 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:55.432438 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:28:55.646842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:55.646807 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:28:55.647095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:55.647053 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" podUID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" containerName="payload-processing" containerID="cri-o://7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8" gracePeriod=30 Oct 30 01:28:55.693282 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:55.693252 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:28:55.693677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:55.693649 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" podUID="977c9c97-6d79-486a-8402-d16f16925b1c" containerName="payload-pre-processing" containerID="cri-o://1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a" gracePeriod=30 Oct 30 01:28:56.027919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.025181 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:28:56.191124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.188391 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\" (UniqueName: \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\") pod \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " Oct 30 01:28:56.191124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.188768 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\") pod \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\" (UID: \"7070c53f-fbfd-4a6a-8e44-fac0064c300c\") " Oct 30 01:28:56.211402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.204929 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume" pod "7070c53f-fbfd-4a6a-8e44-fac0064c300c" (UID: "7070c53f-fbfd-4a6a-8e44-fac0064c300c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:56.281884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.281257 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484" pod "7070c53f-fbfd-4a6a-8e44-fac0064c300c" (UID: "7070c53f-fbfd-4a6a-8e44-fac0064c300c"). InnerVolumeSpecName "kube-api-access-4v484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:56.297262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.297192 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7070c53f-fbfd-4a6a-8e44-fac0064c300c-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:56.297262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.297231 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v484\" (UniqueName: \"kubernetes.io/projected/7070c53f-fbfd-4a6a-8e44-fac0064c300c-kube-api-access-4v484\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:56.352238 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.352164 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 01:28:56.352725 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.352685 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" containerID="cri-o://abe6b41637f8d963720ee5236fc18776d522bf3ba394b7d26079cb745e4b21f4" gracePeriod=30 Oct 30 01:28:56.396548 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.373238 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:28:56.503161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.503078 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\" (UniqueName: \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\") pod \"977c9c97-6d79-486a-8402-d16f16925b1c\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " Oct 30 01:28:56.503798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.503777 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\") pod \"977c9c97-6d79-486a-8402-d16f16925b1c\" (UID: \"977c9c97-6d79-486a-8402-d16f16925b1c\") " Oct 30 01:28:56.505593 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.505571 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume" pod "977c9c97-6d79-486a-8402-d16f16925b1c" (UID: "977c9c97-6d79-486a-8402-d16f16925b1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:56.555112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.549133 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7" pod "977c9c97-6d79-486a-8402-d16f16925b1c" (UID: "977c9c97-6d79-486a-8402-d16f16925b1c"). InnerVolumeSpecName "kube-api-access-4pcf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:56.606579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.606539 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcf7\" (UniqueName: \"kubernetes.io/projected/977c9c97-6d79-486a-8402-d16f16925b1c-kube-api-access-4pcf7\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:56.606579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.606572 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977c9c97-6d79-486a-8402-d16f16925b1c-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:28:56.795670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.795580 5548 generic.go:356] "Generic (PLEG): container finished" podID="977c9c97-6d79-486a-8402-d16f16925b1c" containerID="1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a" exitCode=0 Oct 30 01:28:56.795670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.795653 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" Oct 30 01:28:56.795952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.795664 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" event={"ID":"977c9c97-6d79-486a-8402-d16f16925b1c","Type":"ContainerDied","Data":"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a"} Oct 30 01:28:56.795952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.795735 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z" event={"ID":"977c9c97-6d79-486a-8402-d16f16925b1c","Type":"ContainerDied","Data":"bbc5372aefb355d8fe15d38b88be801a9173aac34d165be6b7382ae7b1fd7176"} Oct 30 01:28:56.795952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.795766 5548 scope.go:122] "RemoveContainer" containerID="1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a" Oct 30 01:28:56.797728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.797587 5548 generic.go:356] "Generic (PLEG): container finished" podID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" containerID="7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8" exitCode=0 Oct 30 01:28:56.797852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.797799 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" event={"ID":"7070c53f-fbfd-4a6a-8e44-fac0064c300c","Type":"ContainerDied","Data":"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8"} Oct 30 01:28:56.797979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.797845 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" event={"ID":"7070c53f-fbfd-4a6a-8e44-fac0064c300c","Type":"ContainerDied","Data":"89601834ac02d4219df35aaff268c33b11193322d0ef55cf5605a59249a20d8a"} Oct 30 01:28:56.797979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.797929 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw" Oct 30 01:28:56.821132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.810575 5548 scope.go:122] "RemoveContainer" containerID="1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a" Oct 30 01:28:56.821132 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:56.810980 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a\": container with ID starting with 1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a not found: ID does not exist" containerID="1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a" Oct 30 01:28:56.821132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.811024 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a"} err="failed to get container status \"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a\": rpc error: code = NotFound desc = could not find container \"1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a\": container with ID starting with 1c35e3578729e3c9d18015532fdd252b36939f0d2d5168ed60b6c60c124c9d8a not found: ID does not exist" Oct 30 01:28:56.821132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.811048 5548 scope.go:122] "RemoveContainer" containerID="7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8" Oct 30 01:28:56.823793 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.823758 5548 scope.go:122] "RemoveContainer" containerID="7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8" Oct 30 01:28:56.825023 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:28:56.824208 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8\": container with ID starting with 7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8 not found: ID does not exist" containerID="7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8" Oct 30 01:28:56.825023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.824256 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8"} err="failed to get container status \"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8\": rpc error: code = NotFound desc = could not find container \"7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8\": container with ID starting with 7057e422f151387eca7a333a5d3c16493dc9ccc2f8553b41e1d3ce6845106af8 not found: ID does not exist" Oct 30 01:28:56.856716 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.856678 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:28:56.868021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.867988 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f0263-56bc5d9dff-wnw6z"] Oct 30 01:28:56.892144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.892117 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:28:56.899786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:56.899747 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f0263-5748487fcb-ttphw"] Oct 30 01:28:57.266711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:57.266679 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" path="/var/lib/kubelet/pods/7070c53f-fbfd-4a6a-8e44-fac0064c300c/volumes" Oct 30 01:28:57.267175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:57.267161 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977c9c97-6d79-486a-8402-d16f16925b1c" path="/var/lib/kubelet/pods/977c9c97-6d79-486a-8402-d16f16925b1c/volumes" Oct 30 01:28:57.946850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:57.946815 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:28:57.947309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:57.947089 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" containerID="cri-o://bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821" gracePeriod=30 Oct 30 01:28:58.768768 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:58.768734 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 01:28:58.769041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:58.768981 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" containerID="cri-o://bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31" gracePeriod=30 Oct 30 01:28:59.272325 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.272292 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:28:59.273803 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.272549 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" containerID="cri-o://a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43" gracePeriod=30 Oct 30 01:28:59.577673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.573717 5548 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:28:59.577673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.573781 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:28:59.635885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.635005 5548 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:28:59.635885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:28:59.635073 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:29:01.852441 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:01.852403 5548 generic.go:356] "Generic (PLEG): container finished" podID="500a5bc0-29ab-4568-a204-370d70d94966" containerID="abe6b41637f8d963720ee5236fc18776d522bf3ba394b7d26079cb745e4b21f4" exitCode=0 Oct 30 01:29:01.852890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:01.852474 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" event={"ID":"500a5bc0-29ab-4568-a204-370d70d94966","Type":"ContainerDied","Data":"abe6b41637f8d963720ee5236fc18776d522bf3ba394b7d26079cb745e4b21f4"} Oct 30 01:29:02.055542 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.055518 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 01:29:02.153839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.153804 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.153945 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.153991 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154038 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154088 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:02.154227 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154123 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154226 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154309 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154347 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154377 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\") pod \"500a5bc0-29ab-4568-a204-370d70d94966\" (UID: \"500a5bc0-29ab-4568-a204-370d70d94966\") " Oct 30 01:29:02.154654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154413 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:02.155203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154740 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:02.155203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154923 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:02.155203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.154982 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:02.155620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.155397 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.155620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.155419 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.155620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.155434 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500a5bc0-29ab-4568-a204-370d70d94966-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.155620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.155447 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.155620 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.155460 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.157190 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.157153 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:02.157407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.157379 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:02.158084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.158046 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "kube-api-access-9v6dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:02.158824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.158802 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token" pod "500a5bc0-29ab-4568-a204-370d70d94966" (UID: "500a5bc0-29ab-4568-a204-370d70d94966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:02.257511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.257468 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500a5bc0-29ab-4568-a204-370d70d94966-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.257511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.257506 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v6dq\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-kube-api-access-9v6dq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.257764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.257521 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500a5bc0-29ab-4568-a204-370d70d94966-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.257764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.257533 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500a5bc0-29ab-4568-a204-370d70d94966-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.861815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.861788 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" Oct 30 01:29:02.862268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.861814 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd" event={"ID":"500a5bc0-29ab-4568-a204-370d70d94966","Type":"ContainerDied","Data":"1b5361151bae01b5740e361485a80476a373697e249c218180c5e939a1ff0535"} Oct 30 01:29:02.862268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.861898 5548 scope.go:122] "RemoveContainer" containerID="abe6b41637f8d963720ee5236fc18776d522bf3ba394b7d26079cb745e4b21f4" Oct 30 01:29:02.919680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.919649 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 01:29:02.928896 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:02.928853 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-588vd"] Oct 30 01:29:03.000047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.000013 5548 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:29:03.000355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.000092 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 01:29:03.280123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.280083 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500a5bc0-29ab-4568-a204-370d70d94966" path="/var/lib/kubelet/pods/500a5bc0-29ab-4568-a204-370d70d94966/volumes" Oct 30 01:29:03.377460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.377435 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:29:03.503785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.503718 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.504536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.504492 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.505363 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505136 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.505547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505028 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:03.505629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505239 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:03.505629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505400 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:03.505810 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505787 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.505974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.505951 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:03.506184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.506158 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.506322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.506209 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.506322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.506242 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.506456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.506388 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.506797 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.506773 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\") pod \"c379f8c4-ecb3-4241-8867-25faf70250a2\" (UID: \"c379f8c4-ecb3-4241-8867-25faf70250a2\") " Oct 30 01:29:03.509006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.507553 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.509006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.507587 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.509006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.507602 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.509006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.507616 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.509006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.508281 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:03.522229 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.522194 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:03.525290 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.525231 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:03.526370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.526338 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:03.529679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.529650 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q" pod "c379f8c4-ecb3-4241-8867-25faf70250a2" (UID: "c379f8c4-ecb3-4241-8867-25faf70250a2"). InnerVolumeSpecName "kube-api-access-dqw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:03.607646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.607611 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:29:03.607917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.607838 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" containerID="cri-o://0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3" gracePeriod=30 Oct 30 01:29:03.608780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.608754 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.608780 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.608780 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqw7q\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-kube-api-access-dqw7q\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.608960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.608790 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.608960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.608799 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c379f8c4-ecb3-4241-8867-25faf70250a2-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.608960 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.608811 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c379f8c4-ecb3-4241-8867-25faf70250a2-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:03.941899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.941827 5548 generic.go:356] "Generic (PLEG): container finished" podID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerID="bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821" exitCode=0 Oct 30 01:29:03.942355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.941979 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" Oct 30 01:29:03.942355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.941989 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" event={"ID":"c379f8c4-ecb3-4241-8867-25faf70250a2","Type":"ContainerDied","Data":"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821"} Oct 30 01:29:03.942355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.942029 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b" event={"ID":"c379f8c4-ecb3-4241-8867-25faf70250a2","Type":"ContainerDied","Data":"8602d6b08fa60f8e1b81496a783d81ac64de0e175e0f8fe658766d781a75880b"} Oct 30 01:29:03.942355 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.942053 5548 scope.go:122] "RemoveContainer" containerID="bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821" Oct 30 01:29:03.955566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.955544 5548 scope.go:122] "RemoveContainer" containerID="bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821" Oct 30 01:29:03.956430 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:03.956342 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821\": container with ID starting with bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821 not found: ID does not exist" containerID="bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821" Oct 30 01:29:03.956530 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.956444 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821"} err="failed to get container status \"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821\": rpc error: code = NotFound desc = could not find container \"bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821\": container with ID starting with bddac23c6f14604fa805fc936ef0cf02f02d37df46b09a7fb989a39060fc3821 not found: ID does not exist" Oct 30 01:29:03.991268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:03.991236 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:29:04.002456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.002425 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-57z4b"] Oct 30 01:29:04.141599 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.141566 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 01:29:04.158020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.157982 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158094 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158132 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158180 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158366 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158207 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158366 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158336 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158387 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158427 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158463 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\") pod \"f7989d88-bade-4470-96c5-9c2e07342c5a\" (UID: \"f7989d88-bade-4470-96c5-9c2e07342c5a\") " Oct 30 01:29:04.158661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158550 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:04.158661 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158555 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.158941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158917 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.159053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.158947 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.159217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.159182 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7989d88-bade-4470-96c5-9c2e07342c5a-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.159217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.159212 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.159362 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.159227 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.159362 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.159240 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.159362 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.159185 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.160848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.160814 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.161132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.161111 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "kube-api-access-cxkxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.161395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.161375 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.161491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.161400 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo" pod "f7989d88-bade-4470-96c5-9c2e07342c5a" (UID: "f7989d88-bade-4470-96c5-9c2e07342c5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:04.260387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.260357 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.260387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.260384 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.260387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.260396 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.260689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.260408 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7989d88-bade-4470-96c5-9c2e07342c5a-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.260689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.260422 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxkxj\" (UniqueName: \"kubernetes.io/projected/f7989d88-bade-4470-96c5-9c2e07342c5a-kube-api-access-cxkxj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.527217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.527190 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:29:04.565541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565508 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.565721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565600 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.565721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565635 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565802 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565838 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565918 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.565991 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566021 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.566350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566109 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566144 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\") pod \"8df2711e-8717-4dd7-85a6-b750c51ba747\" (UID: \"8df2711e-8717-4dd7-85a6-b750c51ba747\") " Oct 30 01:29:04.566350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566263 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.566350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566287 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:04.566562 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566457 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.566659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566621 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.566934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566910 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.566934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566935 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.567079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566950 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.567079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566964 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df2711e-8717-4dd7-85a6-b750c51ba747-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.567079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.566976 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.569186 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.569161 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.569363 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.569344 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.569455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.569428 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:04.570006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.569979 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j" pod "8df2711e-8717-4dd7-85a6-b750c51ba747" (UID: "8df2711e-8717-4dd7-85a6-b750c51ba747"). InnerVolumeSpecName "kube-api-access-mhc8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.668683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.668656 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhc8j\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-kube-api-access-mhc8j\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.668683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.668683 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.668903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.668693 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.668903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.668701 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df2711e-8717-4dd7-85a6-b750c51ba747-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.952276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.952174 5548 generic.go:356] "Generic (PLEG): container finished" podID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerID="bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31" exitCode=0 Oct 30 01:29:04.952276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.952233 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" Oct 30 01:29:04.952939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.952321 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" event={"ID":"f7989d88-bade-4470-96c5-9c2e07342c5a","Type":"ContainerDied","Data":"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31"} Oct 30 01:29:04.952939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.952372 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf" event={"ID":"f7989d88-bade-4470-96c5-9c2e07342c5a","Type":"ContainerDied","Data":"8a6c5fa5d186f9bc164533c1be5c9917c28b627074fc194b4300b05c55b7cc3f"} Oct 30 01:29:04.952939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.952402 5548 scope.go:122] "RemoveContainer" containerID="bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31" Oct 30 01:29:04.954409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.954379 5548 generic.go:356] "Generic (PLEG): container finished" podID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerID="a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43" exitCode=0 Oct 30 01:29:04.954555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.954452 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" event={"ID":"8df2711e-8717-4dd7-85a6-b750c51ba747","Type":"ContainerDied","Data":"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43"} Oct 30 01:29:04.954555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.954478 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" Oct 30 01:29:04.954555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.954498 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86" event={"ID":"8df2711e-8717-4dd7-85a6-b750c51ba747","Type":"ContainerDied","Data":"4d511a8852b43c2d2ec6a4bc57ac0ab4fa28b3d6f6442103e14b2270c4ebe765"} Oct 30 01:29:04.967970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.967946 5548 scope.go:122] "RemoveContainer" containerID="bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31" Oct 30 01:29:04.968342 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:04.968316 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31\": container with ID starting with bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31 not found: ID does not exist" containerID="bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31" Oct 30 01:29:04.968442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.968353 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31"} err="failed to get container status \"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31\": rpc error: code = NotFound desc = could not find container \"bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31\": container with ID starting with bceb7242deb4d1b1fdbc75c3d82212afab9906c6ec95075b23adbba3651f2d31 not found: ID does not exist" Oct 30 01:29:04.968442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.968379 5548 scope.go:122] "RemoveContainer" containerID="a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43" Oct 30 01:29:04.994200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.994180 5548 scope.go:122] "RemoveContainer" containerID="a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43" Oct 30 01:29:04.994501 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:04.994478 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43\": container with ID starting with a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43 not found: ID does not exist" containerID="a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43" Oct 30 01:29:04.994567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:04.994513 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43"} err="failed to get container status \"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43\": rpc error: code = NotFound desc = could not find container \"a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43\": container with ID starting with a48fc0a00456228220bab09ab37a5d4594374915a0a7325e2fa928be14bc9a43 not found: ID does not exist" Oct 30 01:29:05.021407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.021378 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 01:29:05.030817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.030790 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-tb6zf"] Oct 30 01:29:05.059986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.059953 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:29:05.065978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.065949 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-hjw86"] Oct 30 01:29:05.265569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.265476 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" path="/var/lib/kubelet/pods/8df2711e-8717-4dd7-85a6-b750c51ba747/volumes" Oct 30 01:29:05.266141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.266120 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" path="/var/lib/kubelet/pods/c379f8c4-ecb3-4241-8867-25faf70250a2/volumes" Oct 30 01:29:05.266642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:05.266615 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" path="/var/lib/kubelet/pods/f7989d88-bade-4470-96c5-9c2e07342c5a/volumes" Oct 30 01:29:08.883906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.883816 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:29:08.937715 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.937684 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.937934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.937749 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.937934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.937822 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.937934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.937922 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938007 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:08.938133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938046 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938094 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938167 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938252 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938313 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\") pod \"5109d3d7-65d8-4e32-935b-b1813d768a65\" (UID: \"5109d3d7-65d8-4e32-935b-b1813d768a65\") " Oct 30 01:29:08.938415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938165 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:08.938415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938313 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:08.938589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.938542 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:08.939342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.939312 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:08.939342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.939342 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109d3d7-65d8-4e32-935b-b1813d768a65-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:08.939534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.939356 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:08.939534 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.939368 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:08.941759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.941730 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:08.949590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.945989 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:08.949590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.946012 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:08.949590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.946036 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:08.949590 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.946058 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr" pod "5109d3d7-65d8-4e32-935b-b1813d768a65" (UID: "5109d3d7-65d8-4e32-935b-b1813d768a65"). InnerVolumeSpecName "kube-api-access-kx4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:08.993789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.993755 5548 generic.go:356] "Generic (PLEG): container finished" podID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerID="0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3" exitCode=0 Oct 30 01:29:08.993987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.993817 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" Oct 30 01:29:08.993987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.993851 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" event={"ID":"5109d3d7-65d8-4e32-935b-b1813d768a65","Type":"ContainerDied","Data":"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3"} Oct 30 01:29:08.993987 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.993966 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5" event={"ID":"5109d3d7-65d8-4e32-935b-b1813d768a65","Type":"ContainerDied","Data":"67e287deca0556554edf1300a8f2d84ad728f48cd61d72d8984597827cc4414c"} Oct 30 01:29:08.994133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:08.993997 5548 scope.go:122] "RemoveContainer" containerID="0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3" Oct 30 01:29:09.013844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.013817 5548 scope.go:122] "RemoveContainer" containerID="0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3" Oct 30 01:29:09.014203 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:09.014184 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3\": container with ID starting with 0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3 not found: ID does not exist" containerID="0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3" Oct 30 01:29:09.014275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.014214 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3"} err="failed to get container status \"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3\": rpc error: code = NotFound desc = could not find container \"0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3\": container with ID starting with 0d28d945211c10e52d2ea93bf22c779c8dd3a28381c7a4776d295565415c43a3 not found: ID does not exist" Oct 30 01:29:09.040832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.040797 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:09.040832 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.040833 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:09.041052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.040848 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4rr\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-kube-api-access-kx4rr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:09.041052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.040881 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109d3d7-65d8-4e32-935b-b1813d768a65-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:09.041052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.040894 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109d3d7-65d8-4e32-935b-b1813d768a65-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:09.043198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.043176 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:29:09.061666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.061633 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-7gsf5"] Oct 30 01:29:09.266278 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:09.266248 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" path="/var/lib/kubelet/pods/5109d3d7-65d8-4e32-935b-b1813d768a65/volumes" Oct 30 01:29:18.808001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.807963 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:29:18.808443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.808198 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" containerID="cri-o://b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc" gracePeriod=30 Oct 30 01:29:18.826084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.826050 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:29:18.826340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.826309 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" containerID="cri-o://6411f28a7634d3124d0bf29331ad2c7802acadb07b083bbc43e529b4f8b7acd5" gracePeriod=30 Oct 30 01:29:18.829401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.829364 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:29:18.829690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:18.829663 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" containerID="cri-o://bb82907693e1e4bf08824c6c28b97e05bf302393f0b61e8e89147e1a8fcb9b55" gracePeriod=30 Oct 30 01:29:24.127037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.127002 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:29:24.154276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.153853 5548 generic.go:356] "Generic (PLEG): container finished" podID="0e41f202-179e-47fc-b776-4c744541606f" containerID="b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc" exitCode=0 Oct 30 01:29:24.154276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.153960 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" Oct 30 01:29:24.154276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.154064 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" event={"ID":"0e41f202-179e-47fc-b776-4c744541606f","Type":"ContainerDied","Data":"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc"} Oct 30 01:29:24.154276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.154117 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt" event={"ID":"0e41f202-179e-47fc-b776-4c744541606f","Type":"ContainerDied","Data":"d2ea1850d702d1f04d1587ee4e7601ca295cfcda2d5d7d4c1be0615255c94755"} Oct 30 01:29:24.154276 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.154149 5548 scope.go:122] "RemoveContainer" containerID="b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc" Oct 30 01:29:24.157239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.157213 5548 generic.go:356] "Generic (PLEG): container finished" podID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerID="bb82907693e1e4bf08824c6c28b97e05bf302393f0b61e8e89147e1a8fcb9b55" exitCode=0 Oct 30 01:29:24.157356 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.157322 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" event={"ID":"3740ffd4-a844-4c13-8914-ec81a7b5c129","Type":"ContainerDied","Data":"bb82907693e1e4bf08824c6c28b97e05bf302393f0b61e8e89147e1a8fcb9b55"} Oct 30 01:29:24.159185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.159159 5548 generic.go:356] "Generic (PLEG): container finished" podID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerID="6411f28a7634d3124d0bf29331ad2c7802acadb07b083bbc43e529b4f8b7acd5" exitCode=0 Oct 30 01:29:24.159552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.159528 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" event={"ID":"04658015-a02c-4f70-a006-71a6e1f10a5f","Type":"ContainerDied","Data":"6411f28a7634d3124d0bf29331ad2c7802acadb07b083bbc43e529b4f8b7acd5"} Oct 30 01:29:24.159638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.159600 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" event={"ID":"04658015-a02c-4f70-a006-71a6e1f10a5f","Type":"ContainerDied","Data":"78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b"} Oct 30 01:29:24.159638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.159630 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b0f114d96f4faf6e124bf1260c88520702fdf291d52c29d1a4e45481149b6b" Oct 30 01:29:24.175608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.175582 5548 scope.go:122] "RemoveContainer" containerID="b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc" Oct 30 01:29:24.175933 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:24.175906 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc\": container with ID starting with b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc not found: ID does not exist" containerID="b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc" Oct 30 01:29:24.176001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.175949 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc"} err="failed to get container status \"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc\": rpc error: code = NotFound desc = could not find container \"b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc\": container with ID starting with b1bdd474a36da01a6144bfb5a1d2493d8510e6b23bbe1e508dc994ac6c5a2fcc not found: ID does not exist" Oct 30 01:29:24.176455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.176437 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:29:24.198627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198598 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.198789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198651 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.198789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198691 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.198789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198738 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.198789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198775 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.199079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198824 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.199079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.198934 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.199079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.199003 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.199291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.199163 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.200582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.199632 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.200582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.199665 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.200582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.199980 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.200582 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.200414 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.201060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.200584 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.201060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.200935 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.201281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.201163 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.201457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.201439 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.201618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.201601 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.201946 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.201888 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:29:24.202038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.201980 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.202159 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.202138 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.202289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.202269 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.202401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.202385 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\") pod \"04658015-a02c-4f70-a006-71a6e1f10a5f\" (UID: \"04658015-a02c-4f70-a006-71a6e1f10a5f\") " Oct 30 01:29:24.202526 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.202510 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.202679 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.202663 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\") pod \"0e41f202-179e-47fc-b776-4c744541606f\" (UID: \"0e41f202-179e-47fc-b776-4c744541606f\") " Oct 30 01:29:24.203432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.203405 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "kube-api-access-5jz8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.203682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.203630 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.203810 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.203790 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.203906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.203838 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.204212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204102 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.204212 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204176 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.204382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204368 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.204668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204645 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.204791 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204776 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.204931 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.204917 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205035 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205023 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205182 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205170 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e41f202-179e-47fc-b776-4c744541606f-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205274 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205348 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jz8j\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-kube-api-access-5jz8j\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205368 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205382 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205395 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04658015-a02c-4f70-a006-71a6e1f10a5f-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205408 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04658015-a02c-4f70-a006-71a6e1f10a5f-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205422 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e41f202-179e-47fc-b776-4c744541606f-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205435 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.205762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205457 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.205825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.205798 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.206274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.206249 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo" pod "04658015-a02c-4f70-a006-71a6e1f10a5f" (UID: "04658015-a02c-4f70-a006-71a6e1f10a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.207514 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.207490 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.213131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.213086 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk" pod "0e41f202-179e-47fc-b776-4c744541606f" (UID: "0e41f202-179e-47fc-b776-4c744541606f"). InnerVolumeSpecName "kube-api-access-wg5nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.251750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.251721 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306360 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306453 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306517 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306555 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306629 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306670 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306715 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306823 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.307345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.306884 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\") pod \"3740ffd4-a844-4c13-8914-ec81a7b5c129\" (UID: \"3740ffd4-a844-4c13-8914-ec81a7b5c129\") " Oct 30 01:29:24.308030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.307838 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.308030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.307907 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.308030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.307922 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.308030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308024 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.308516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308493 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.308743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308722 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308747 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308759 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3740ffd4-a844-4c13-8914-ec81a7b5c129-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308770 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04658015-a02c-4f70-a006-71a6e1f10a5f-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308782 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308793 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e41f202-179e-47fc-b776-4c744541606f-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308804 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5nk\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-kube-api-access-wg5nk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308816 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e41f202-179e-47fc-b776-4c744541606f-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308829 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.308837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.308841 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.310265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.310241 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.311106 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.311080 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.311240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.311224 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "kube-api-access-s72ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.311311 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.311276 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token" pod "3740ffd4-a844-4c13-8914-ec81a7b5c129" (UID: "3740ffd4-a844-4c13-8914-ec81a7b5c129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.410338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.410307 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72ln\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-kube-api-access-s72ln\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.410338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.410338 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.410338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.410349 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.410709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.410357 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3740ffd4-a844-4c13-8914-ec81a7b5c129-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.501941 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.501905 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:29:24.505442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:24.505415 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-9srvt"] Oct 30 01:29:25.170428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.170399 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" Oct 30 01:29:25.170943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.170428 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl" event={"ID":"3740ffd4-a844-4c13-8914-ec81a7b5c129","Type":"ContainerDied","Data":"9dee578058aa1f73855520763bd5da84757430e181947c12a95d92a4c52d5571"} Oct 30 01:29:25.170943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.170492 5548 scope.go:122] "RemoveContainer" containerID="bb82907693e1e4bf08824c6c28b97e05bf302393f0b61e8e89147e1a8fcb9b55" Oct 30 01:29:25.170943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.170661 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp" Oct 30 01:29:25.237019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.236982 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:29:25.247288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.247236 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-mknfl"] Oct 30 01:29:25.267603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.267566 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e41f202-179e-47fc-b776-4c744541606f" path="/var/lib/kubelet/pods/0e41f202-179e-47fc-b776-4c744541606f/volumes" Oct 30 01:29:25.268138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.268119 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" path="/var/lib/kubelet/pods/3740ffd4-a844-4c13-8914-ec81a7b5c129/volumes" Oct 30 01:29:25.274275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.274245 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:29:25.278402 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:25.278377 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-424hp"] Oct 30 01:29:25.428298 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:25.428207 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:29:27.264776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:27.264742 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" path="/var/lib/kubelet/pods/04658015-a02c-4f70-a006-71a6e1f10a5f/volumes" Oct 30 01:29:28.702162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:28.702121 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:29:28.702659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:28.702337 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" containerID="cri-o://9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4" gracePeriod=30 Oct 30 01:29:30.873909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:30.873878 5548 scope.go:122] "RemoveContainer" containerID="6411f28a7634d3124d0bf29331ad2c7802acadb07b083bbc43e529b4f8b7acd5" Oct 30 01:29:33.988463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:33.988406 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:29:34.075064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075031 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075099 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075154 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075190 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075221 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075270 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075338 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075388 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:34.075520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075426 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075482 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\") pod \"3a34cd0e-2853-426b-a535-ae50e90bf222\" (UID: \"3a34cd0e-2853-426b-a535-ae50e90bf222\") " Oct 30 01:29:34.075808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075529 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:34.075808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075609 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:34.075998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.075969 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:34.076123 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.076103 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.076193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.076134 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.076193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.076148 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.076193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.076162 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.076193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.076104 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:34.077569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.077541 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:34.077569 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.077548 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:34.078369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.078345 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:34.078468 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.078450 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6" pod "3a34cd0e-2853-426b-a535-ae50e90bf222" (UID: "3a34cd0e-2853-426b-a535-ae50e90bf222"). InnerVolumeSpecName "kube-api-access-52hz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:34.177373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.177343 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hz6\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-kube-api-access-52hz6\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.177373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.177368 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.177373 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.177377 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a34cd0e-2853-426b-a535-ae50e90bf222-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.177606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.177385 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.177606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.177395 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a34cd0e-2853-426b-a535-ae50e90bf222-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:34.238242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.238197 5548 generic.go:356] "Generic (PLEG): container finished" podID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerID="9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4" exitCode=0 Oct 30 01:29:34.238399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.238249 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" Oct 30 01:29:34.238399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.238289 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" event={"ID":"3a34cd0e-2853-426b-a535-ae50e90bf222","Type":"ContainerDied","Data":"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4"} Oct 30 01:29:34.238399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.238345 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d" event={"ID":"3a34cd0e-2853-426b-a535-ae50e90bf222","Type":"ContainerDied","Data":"9deca290fd26d7e841219a889c8315e194a7081bef96cf2cfcb74da5e1d9f31b"} Oct 30 01:29:34.238399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.238369 5548 scope.go:122] "RemoveContainer" containerID="9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4" Oct 30 01:29:34.250409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.250390 5548 scope.go:122] "RemoveContainer" containerID="9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4" Oct 30 01:29:34.250727 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:34.250709 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4\": container with ID starting with 9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4 not found: ID does not exist" containerID="9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4" Oct 30 01:29:34.250794 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.250736 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4"} err="failed to get container status \"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4\": rpc error: code = NotFound desc = could not find container \"9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4\": container with ID starting with 9ab88c359eeea052080ec51c42c281ba05c73c50c6c180ccfecca322f766fff4 not found: ID does not exist" Oct 30 01:29:34.286463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.286433 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:29:34.290250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:34.290229 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-64f472-openshift-default-5b7c8dc98c-9dp4d"] Oct 30 01:29:35.265455 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:35.265417 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" path="/var/lib/kubelet/pods/3a34cd0e-2853-426b-a535-ae50e90bf222/volumes" Oct 30 01:29:39.282170 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.282127 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" containerID="cri-o://0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df" gracePeriod=45 Oct 30 01:29:39.558815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.558789 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:29:39.663515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663297 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.663731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663552 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.663731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663584 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.663731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663684 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\" (UniqueName: \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.663731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663739 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.664006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.663986 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.664080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.664070 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\") pod \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\" (UID: \"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e\") " Oct 30 01:29:39.664543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.664523 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.664769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.664754 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.664988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.664969 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.666840 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.666817 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.667448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.667426 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:39.668020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.668000 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt" pod "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" (UID: "a3bf8af7-6f75-4cc1-89d1-64611c87ea4e"). InnerVolumeSpecName "kube-api-access-c6tkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:39.766957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.766783 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.766957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.766846 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.766957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.766882 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.766957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.766895 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tkt\" (UniqueName: \"kubernetes.io/projected/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-kube-api-access-c6tkt\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.766957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:39.766920 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:40.290986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.290945 5548 generic.go:356] "Generic (PLEG): container finished" podID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerID="0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df" exitCode=0 Oct 30 01:29:40.291435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.291002 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" Oct 30 01:29:40.291435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.290997 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" event={"ID":"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e","Type":"ContainerDied","Data":"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df"} Oct 30 01:29:40.291435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.291066 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4" event={"ID":"a3bf8af7-6f75-4cc1-89d1-64611c87ea4e","Type":"ContainerDied","Data":"28ec40137e2e0a077767e312453461130c1d53b8d4e062499455ddc30c56ecd5"} Oct 30 01:29:40.291435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.291090 5548 scope.go:122] "RemoveContainer" containerID="0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df" Oct 30 01:29:40.310241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.310217 5548 scope.go:122] "RemoveContainer" containerID="0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df" Oct 30 01:29:40.310558 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:40.310536 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df\": container with ID starting with 0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df not found: ID does not exist" containerID="0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df" Oct 30 01:29:40.310645 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.310570 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df"} err="failed to get container status \"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df\": rpc error: code = NotFound desc = could not find container \"0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df\": container with ID starting with 0f7c468739073c3042bde67493c8bcc39b88d939990130fb57922c41ea8007df not found: ID does not exist" Oct 30 01:29:40.336934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.336901 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 01:29:40.346536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:40.346512 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/simulator-w2-fd3281-kserve-66486987bf-pfss4"] Oct 30 01:29:41.264209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:41.264179 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" path="/var/lib/kubelet/pods/a3bf8af7-6f75-4cc1-89d1-64611c87ea4e/volumes" Oct 30 01:29:44.402017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.401983 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:29:44.402964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.402924 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" containerID="cri-o://b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782" gracePeriod=30 Oct 30 01:29:44.991752 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.991714 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:29:44.992314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.992274 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" containerID="cri-o://652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836" gracePeriod=30 Oct 30 01:29:44.995753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.995724 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:29:44.995932 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:44.995907 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" containerID="cri-o://576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255" gracePeriod=30 Oct 30 01:29:48.527529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:48.527490 5548 patch_prober.go:29] interesting pod/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:29:48.527959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:48.527549 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:29:49.459208 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:49.459169 5548 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde533c29_45f0_49a8_9ee4_36637854c549.slice/crio-b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde533c29_45f0_49a8_9ee4_36637854c549.slice/crio-conmon-b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:29:49.677160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.677108 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:29:49.830635 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830598 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.830834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830650 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.830834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830688 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.830834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830739 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.830834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830781 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.830834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830810 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.831177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.830854 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.831177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831035 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.831177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831070 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\") pod \"de533c29-45f0-49a8-9ee4-36637854c549\" (UID: \"de533c29-45f0-49a8-9ee4-36637854c549\") " Oct 30 01:29:49.831177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831073 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:49.831177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831086 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:49.831474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831193 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:49.831474 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831316 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:49.831595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831550 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:49.831684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831656 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.831684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831681 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de533c29-45f0-49a8-9ee4-36637854c549-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.832001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831691 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.832001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831700 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.832001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.831711 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.833173 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.833140 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:49.833350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.833330 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:49.833552 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.833535 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "kube-api-access-tbd65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:49.833666 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.833647 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token" pod "de533c29-45f0-49a8-9ee4-36637854c549" (UID: "de533c29-45f0-49a8-9ee4-36637854c549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:49.857377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.857347 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 01:29:49.879685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.879652 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 01:29:49.932644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.932553 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de533c29-45f0-49a8-9ee4-36637854c549-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.932644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.932586 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbd65\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-kube-api-access-tbd65\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.932644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.932598 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de533c29-45f0-49a8-9ee4-36637854c549-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:49.932644 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:49.932607 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de533c29-45f0-49a8-9ee4-36637854c549-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.283242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.283215 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:29:50.307371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.307347 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:29:50.379833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.379797 5548 generic.go:356] "Generic (PLEG): container finished" podID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerID="576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255" exitCode=0 Oct 30 01:29:50.380048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.379845 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" Oct 30 01:29:50.380048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.379926 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" event={"ID":"7c0cbbd8-be34-49c9-8b06-645956c95542","Type":"ContainerDied","Data":"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255"} Oct 30 01:29:50.380048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.379996 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm" event={"ID":"7c0cbbd8-be34-49c9-8b06-645956c95542","Type":"ContainerDied","Data":"dcb303ad69af39dd6baa0f89df71b54756a42fea90f69ffbdbf0c7bc0b6bbc28"} Oct 30 01:29:50.380048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.380041 5548 scope.go:122] "RemoveContainer" containerID="576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255" Oct 30 01:29:50.382318 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.382278 5548 generic.go:356] "Generic (PLEG): container finished" podID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerID="652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836" exitCode=0 Oct 30 01:29:50.382443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.382370 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerDied","Data":"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836"} Oct 30 01:29:50.382443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.382435 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" event={"ID":"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40","Type":"ContainerDied","Data":"c092b8e8096317aa364c2ddcb3e1408479efe7b66e620ccf13ca16062e6fcc71"} Oct 30 01:29:50.382637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.382611 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c" Oct 30 01:29:50.384811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.384785 5548 generic.go:356] "Generic (PLEG): container finished" podID="de533c29-45f0-49a8-9ee4-36637854c549" containerID="b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782" exitCode=0 Oct 30 01:29:50.384975 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.384939 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerDied","Data":"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782"} Oct 30 01:29:50.385049 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.384981 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" event={"ID":"de533c29-45f0-49a8-9ee4-36637854c549","Type":"ContainerDied","Data":"e1480cae4d34a1e66b1b6685b1de57e078e16b019de6584f63c5b7ee6b410aa0"} Oct 30 01:29:50.385110 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.385065 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7" Oct 30 01:29:50.393266 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.393249 5548 scope.go:122] "RemoveContainer" containerID="576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255" Oct 30 01:29:50.393535 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:50.393516 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255\": container with ID starting with 576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255 not found: ID does not exist" containerID="576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255" Oct 30 01:29:50.393631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.393542 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255"} err="failed to get container status \"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255\": rpc error: code = NotFound desc = could not find container \"576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255\": container with ID starting with 576af4b338e80caac1b9d88917d9c432042ebf7047d3acddf09eb35a26e28255 not found: ID does not exist" Oct 30 01:29:50.393631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.393556 5548 scope.go:122] "RemoveContainer" containerID="652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836" Oct 30 01:29:50.403817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.403785 5548 scope.go:122] "RemoveContainer" containerID="89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca" Oct 30 01:29:50.413286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.413257 5548 scope.go:122] "RemoveContainer" containerID="652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836" Oct 30 01:29:50.413537 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:50.413519 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836\": container with ID starting with 652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836 not found: ID does not exist" containerID="652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836" Oct 30 01:29:50.413585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.413546 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836"} err="failed to get container status \"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836\": rpc error: code = NotFound desc = could not find container \"652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836\": container with ID starting with 652aadc975f5067f79932c39820b6e2123e08308a90436ef56b0594036c32836 not found: ID does not exist" Oct 30 01:29:50.413585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.413567 5548 scope.go:122] "RemoveContainer" containerID="89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca" Oct 30 01:29:50.413812 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:50.413794 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca\": container with ID starting with 89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca not found: ID does not exist" containerID="89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca" Oct 30 01:29:50.413880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.413819 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca"} err="failed to get container status \"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca\": rpc error: code = NotFound desc = could not find container \"89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca\": container with ID starting with 89cf15df939dac4f83573cfb4e3ca6ae845cb85d137f0d409270e18ebe90e6ca not found: ID does not exist" Oct 30 01:29:50.413880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.413835 5548 scope.go:122] "RemoveContainer" containerID="b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782" Oct 30 01:29:50.423331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.423315 5548 scope.go:122] "RemoveContainer" containerID="62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0" Oct 30 01:29:50.439522 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439499 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.439627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439541 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.439714 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439690 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.439771 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439735 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.439815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439769 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.439815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439798 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.439942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439841 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.439942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439881 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.439942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439895 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439980 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439980 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.439995 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440028 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440056 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440087 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.440452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440147 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440161 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440207 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440272 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.440452 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440402 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440660 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.440735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440392 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440790 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.440898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440885 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\") pod \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\" (UID: \"6bac41a7-ccbe-46f9-9f19-8a6d4559fa40\") " Oct 30 01:29:50.441031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440904 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:50.441031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440916 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\") pod \"7c0cbbd8-be34-49c9-8b06-645956c95542\" (UID: \"7c0cbbd8-be34-49c9-8b06-645956c95542\") " Oct 30 01:29:50.441031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.440925 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441247 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441827 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441847 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441877 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441890 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441901 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441913 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441923 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441935 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0cbbd8-be34-49c9-8b06-645956c95542-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441945 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.441985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.441958 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.442634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.442455 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "kube-api-access-bchw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:50.443796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.443743 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:50.443796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.443764 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:50.443970 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.443839 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "kube-api-access-ctwkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:50.444064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.444041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.444267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.444234 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:50.444773 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.444756 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo" pod "7c0cbbd8-be34-49c9-8b06-645956c95542" (UID: "7c0cbbd8-be34-49c9-8b06-645956c95542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:50.445318 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.445301 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy" pod "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" (UID: "6bac41a7-ccbe-46f9-9f19-8a6d4559fa40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:50.456949 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.456924 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:29:50.460675 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.460621 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-999ff9-openshift-default-64f4c6f67c-cqzk7"] Oct 30 01:29:50.476076 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.476056 5548 scope.go:122] "RemoveContainer" containerID="b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782" Oct 30 01:29:50.476430 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:50.476413 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782\": container with ID starting with b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782 not found: ID does not exist" containerID="b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782" Oct 30 01:29:50.476480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.476440 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782"} err="failed to get container status \"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782\": rpc error: code = NotFound desc = could not find container \"b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782\": container with ID starting with b1f1a0f2b2442a38dcfedfec41c8206d8d6a87f92579e753a600cb4b26aca782 not found: ID does not exist" Oct 30 01:29:50.476480 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.476457 5548 scope.go:122] "RemoveContainer" containerID="62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0" Oct 30 01:29:50.476726 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:50.476707 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0\": container with ID starting with 62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0 not found: ID does not exist" containerID="62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0" Oct 30 01:29:50.476789 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.476732 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0"} err="failed to get container status \"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0\": rpc error: code = NotFound desc = could not find container \"62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0\": container with ID starting with 62b89e2d12a38550317f22ae309efc2fe4b5fa6a7dedbc2b58d99bc1056a59d0 not found: ID does not exist" Oct 30 01:29:50.491003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.490977 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 01:29:50.542718 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542690 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542718 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542715 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctwkp\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-kube-api-access-ctwkp\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542726 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542737 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542746 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542754 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0cbbd8-be34-49c9-8b06-645956c95542-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542762 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.542969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.542769 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchw6\" (UniqueName: \"kubernetes.io/projected/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40-kube-api-access-bchw6\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:50.725042 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.724955 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:29:50.727757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.727737 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-rklxm"] Oct 30 01:29:50.754621 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.754589 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:29:50.760376 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:50.760350 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-add1bb-openshift-default-8bdb47f67-r9w5c"] Oct 30 01:29:51.265245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:51.265208 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" path="/var/lib/kubelet/pods/6bac41a7-ccbe-46f9-9f19-8a6d4559fa40/volumes" Oct 30 01:29:51.265698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:51.265684 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" path="/var/lib/kubelet/pods/7c0cbbd8-be34-49c9-8b06-645956c95542/volumes" Oct 30 01:29:51.266129 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:51.266114 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de533c29-45f0-49a8-9ee4-36637854c549" path="/var/lib/kubelet/pods/de533c29-45f0-49a8-9ee4-36637854c549/volumes" Oct 30 01:29:55.430422 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:29:55.430375 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:29:58.058636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.058601 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:29:58.059137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.058789 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" podUID="bd6b01a5-90fd-4827-9963-834850ee4c65" containerName="maas-api" containerID="cri-o://35a75e3162b3302b86da033ad5f88f30539124d05e84c47849f7883a9532222f" gracePeriod=30 Oct 30 01:29:58.457843 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.457605 5548 generic.go:356] "Generic (PLEG): container finished" podID="bd6b01a5-90fd-4827-9963-834850ee4c65" containerID="35a75e3162b3302b86da033ad5f88f30539124d05e84c47849f7883a9532222f" exitCode=0 Oct 30 01:29:58.457843 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.457777 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" event={"ID":"bd6b01a5-90fd-4827-9963-834850ee4c65","Type":"ContainerDied","Data":"35a75e3162b3302b86da033ad5f88f30539124d05e84c47849f7883a9532222f"} Oct 30 01:29:58.564015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.563978 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:29:58.684549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.684518 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\") pod \"bd6b01a5-90fd-4827-9963-834850ee4c65\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " Oct 30 01:29:58.684715 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.684683 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\") pod \"bd6b01a5-90fd-4827-9963-834850ee4c65\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " Oct 30 01:29:58.684762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.684723 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\") pod \"bd6b01a5-90fd-4827-9963-834850ee4c65\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " Oct 30 01:29:58.684762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.684743 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\" (UniqueName: \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\") pod \"bd6b01a5-90fd-4827-9963-834850ee4c65\" (UID: \"bd6b01a5-90fd-4827-9963-834850ee4c65\") " Oct 30 01:29:58.685163 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.685134 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca" pod "bd6b01a5-90fd-4827-9963-834850ee4c65" (UID: "bd6b01a5-90fd-4827-9963-834850ee4c65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:58.685312 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.685294 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd6b01a5-90fd-4827-9963-834850ee4c65-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:58.686841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.686815 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls" pod "bd6b01a5-90fd-4827-9963-834850ee4c65" (UID: "bd6b01a5-90fd-4827-9963-834850ee4c65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:58.686972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.686931 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs" pod "bd6b01a5-90fd-4827-9963-834850ee4c65" (UID: "bd6b01a5-90fd-4827-9963-834850ee4c65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:58.688318 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.688293 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd" pod "bd6b01a5-90fd-4827-9963-834850ee4c65" (UID: "bd6b01a5-90fd-4827-9963-834850ee4c65"). InnerVolumeSpecName "kube-api-access-8zkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:58.787064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.786986 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:58.787064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.787012 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zkgd\" (UniqueName: \"kubernetes.io/projected/bd6b01a5-90fd-4827-9963-834850ee4c65-kube-api-access-8zkgd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:58.787064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:58.787023 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd6b01a5-90fd-4827-9963-834850ee4c65-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:29:59.465659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:59.465623 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" Oct 30 01:29:59.466108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:59.465648 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl" event={"ID":"bd6b01a5-90fd-4827-9963-834850ee4c65","Type":"ContainerDied","Data":"2e6621766adac81dba85974ea58b9b4f13aad96cdd0d0ce797272a689b99b9a4"} Oct 30 01:29:59.466108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:59.465719 5548 scope.go:122] "RemoveContainer" containerID="35a75e3162b3302b86da033ad5f88f30539124d05e84c47849f7883a9532222f" Oct 30 01:29:59.513746 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:59.513707 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:29:59.523652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:29:59.523591 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-29145c-65f58488c7-rz2xl"] Oct 30 01:30:00.198025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.197993 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:00.198834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198808 5548 cpu_manager.go:414] "containerMap: removing container" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" Oct 30 01:30:00.198834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198834 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198852 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerName="maas-api" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198878 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerName="maas-api" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198897 5548 cpu_manager.go:414] "containerMap: removing container" podUID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" containerName="payload-processing" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198904 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" containerName="payload-processing" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198916 5548 cpu_manager.go:414] "containerMap: removing container" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198924 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198933 5548 cpu_manager.go:414] "containerMap: removing container" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198941 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198951 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198958 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198970 5548 cpu_manager.go:414] "containerMap: removing container" podUID="bd6b01a5-90fd-4827-9963-834850ee4c65" containerName="maas-api" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198976 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6b01a5-90fd-4827-9963-834850ee4c65" containerName="maas-api" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198988 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.198997 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199012 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199019 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199028 5548 cpu_manager.go:414] "containerMap: removing container" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199034 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199052 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" Oct 30 01:30:00.199053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199059 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199080 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199087 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199101 5548 cpu_manager.go:414] "containerMap: removing container" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199108 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199124 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199130 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199139 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199145 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199156 5548 cpu_manager.go:414] "containerMap: removing container" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199163 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199186 5548 cpu_manager.go:414] "containerMap: removing container" podUID="16f9868e-83be-4b11-a343-4b0609e143b5" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199194 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f9868e-83be-4b11-a343-4b0609e143b5" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199206 5548 cpu_manager.go:414] "containerMap: removing container" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199213 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199224 5548 cpu_manager.go:414] "containerMap: removing container" podUID="977c9c97-6d79-486a-8402-d16f16925b1c" containerName="payload-pre-processing" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199231 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977c9c97-6d79-486a-8402-d16f16925b1c" containerName="payload-pre-processing" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199251 5548 cpu_manager.go:414] "containerMap: removing container" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199258 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199270 5548 cpu_manager.go:414] "containerMap: removing container" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199276 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199286 5548 cpu_manager.go:414] "containerMap: removing container" podUID="5b108e37-6075-4bb1-9204-bdf0712a9c05" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199292 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b108e37-6075-4bb1-9204-bdf0712a9c05" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199305 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199314 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199327 5548 cpu_manager.go:414] "containerMap: removing container" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199333 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199478 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f9868e-83be-4b11-a343-4b0609e143b5" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199490 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199502 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="977c9c97-6d79-486a-8402-d16f16925b1c" containerName="payload-pre-processing" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199514 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199522 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3740ffd4-a844-4c13-8914-ec81a7b5c129" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199530 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b108e37-6075-4bb1-9204-bdf0712a9c05" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199539 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199549 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="de533c29-45f0-49a8-9ee4-36637854c549" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199557 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bf8af7-6f75-4cc1-89d1-64611c87ea4e" containerName="main" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199567 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba6f1aa-fc5b-420e-a649-94657c8f6a19" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199579 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="7070c53f-fbfd-4a6a-8e44-fac0064c300c" containerName="payload-processing" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199589 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="500a5bc0-29ab-4568-a204-370d70d94966" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199599 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="04658015-a02c-4f70-a006-71a6e1f10a5f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199609 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd5cd32-44ce-4ab0-8120-49af64d8dc02" containerName="main" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199658 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6b01a5-90fd-4827-9963-834850ee4c65" containerName="maas-api" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199668 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a34cd0e-2853-426b-a535-ae50e90bf222" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199682 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac41a7-ccbe-46f9-9f19-8a6d4559fa40" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199692 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e41f202-179e-47fc-b776-4c744541606f" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199704 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c379f8c4-ecb3-4241-8867-25faf70250a2" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199714 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7989d88-bade-4470-96c5-9c2e07342c5a" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199726 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="5109d3d7-65d8-4e32-935b-b1813d768a65" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199734 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df2711e-8717-4dd7-85a6-b750c51ba747" containerName="istio-proxy" Oct 30 01:30:00.200614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.199741 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0cbbd8-be34-49c9-8b06-645956c95542" containerName="istio-proxy" Oct 30 01:30:00.207196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.207157 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.229917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.229883 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:00.238160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.238137 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:00.238899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.238849 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16525c9-524a-4635-8057-9712f693e64f" containerName="istio-proxy" Oct 30 01:30:00.242442 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.242423 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.258924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.258356 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:00.281029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.280996 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:30:00.304761 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.304731 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.304935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.304891 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.305013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.304990 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7lq\" (UniqueName: \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.305135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.305116 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ps6\" (UniqueName: \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.406490 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.406459 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ps6\" (UniqueName: \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.406648 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.406553 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.406648 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.406605 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.406798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.406732 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7lq\" (UniqueName: \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.407329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.407306 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.407457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.407360 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.427313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.427282 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7lq\" (UniqueName: \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\") pod \"maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.427678 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.427655 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ps6\" (UniqueName: \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\") pod \"maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.529114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.529040 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:00.557364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.557339 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:00.831021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.830989 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:00.832578 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:30:00.832547 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8988d9f8_b11b_454c_b1aa_f859e56d4389.slice/crio-edc942cc1f38a4b66f1a5f6cd971b9e0209cdc802018a617ae8efa1856471601 WatchSource:0}: Error finding container edc942cc1f38a4b66f1a5f6cd971b9e0209cdc802018a617ae8efa1856471601: Status 404 returned error can't find the container with id edc942cc1f38a4b66f1a5f6cd971b9e0209cdc802018a617ae8efa1856471601 Oct 30 01:30:00.899073 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:00.898909 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:00.900080 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:30:00.900056 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edf8627_750a_48a8_a30d_d8106888e2d1.slice/crio-ee01d9697efd18f657e326d5ea561766878e9c542526ffeb49c8b147200b08bf WatchSource:0}: Error finding container ee01d9697efd18f657e326d5ea561766878e9c542526ffeb49c8b147200b08bf: Status 404 returned error can't find the container with id ee01d9697efd18f657e326d5ea561766878e9c542526ffeb49c8b147200b08bf Oct 30 01:30:01.264592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.264559 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6b01a5-90fd-4827-9963-834850ee4c65" path="/var/lib/kubelet/pods/bd6b01a5-90fd-4827-9963-834850ee4c65/volumes" Oct 30 01:30:01.524886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.524574 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerStarted","Data":"dc95b808c932842e0b75ff89e9b170906c32c035342751edc17d087f90d9d308"} Oct 30 01:30:01.524886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.524646 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerStarted","Data":"edc942cc1f38a4b66f1a5f6cd971b9e0209cdc802018a617ae8efa1856471601"} Oct 30 01:30:01.544589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.537444 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerStarted","Data":"d3eb8653888637088d7d8aa9750f8ea01206ca2644511d531ae0c27d8d0fb9d6"} Oct 30 01:30:01.544589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.537509 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerStarted","Data":"ee01d9697efd18f657e326d5ea561766878e9c542526ffeb49c8b147200b08bf"} Oct 30 01:30:01.658257 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.658198 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" podStartSLOduration=1.658182729 podStartE2EDuration="1.658182729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.656530042 +0000 UTC m=+2792.499090635" watchObservedRunningTime="2026-10-30 01:30:01.658182729 +0000 UTC m=+2792.500743322" Oct 30 01:30:01.658575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:01.658553 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" podStartSLOduration=1.658541757 podStartE2EDuration="1.658541757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.604061981 +0000 UTC m=+2792.446622575" watchObservedRunningTime="2026-10-30 01:30:01.658541757 +0000 UTC m=+2792.501102352" Oct 30 01:30:02.576885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.576312 5548 generic.go:356] "Generic (PLEG): container finished" podID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerID="d3eb8653888637088d7d8aa9750f8ea01206ca2644511d531ae0c27d8d0fb9d6" exitCode=60 Oct 30 01:30:02.576885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.576674 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerDied","Data":"d3eb8653888637088d7d8aa9750f8ea01206ca2644511d531ae0c27d8d0fb9d6"} Oct 30 01:30:02.577465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.577194 5548 scope.go:122] "RemoveContainer" containerID="d3eb8653888637088d7d8aa9750f8ea01206ca2644511d531ae0c27d8d0fb9d6" Oct 30 01:30:02.582332 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.582302 5548 generic.go:356] "Generic (PLEG): container finished" podID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerID="dc95b808c932842e0b75ff89e9b170906c32c035342751edc17d087f90d9d308" exitCode=60 Oct 30 01:30:02.582579 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.582553 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerDied","Data":"dc95b808c932842e0b75ff89e9b170906c32c035342751edc17d087f90d9d308"} Oct 30 01:30:02.583037 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:02.583008 5548 scope.go:122] "RemoveContainer" containerID="dc95b808c932842e0b75ff89e9b170906c32c035342751edc17d087f90d9d308" Oct 30 01:30:03.590702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:03.590671 5548 generic.go:356] "Generic (PLEG): container finished" podID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerID="48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51" exitCode=60 Oct 30 01:30:03.591125 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:03.590748 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerDied","Data":"48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51"} Oct 30 01:30:03.591125 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:03.590807 5548 scope.go:122] "RemoveContainer" containerID="dc95b808c932842e0b75ff89e9b170906c32c035342751edc17d087f90d9d308" Oct 30 01:30:03.591125 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:03.591114 5548 scope.go:122] "RemoveContainer" containerID="48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51" Oct 30 01:30:03.591466 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:03.591440 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c_odh-ai-gateway-infra(8988d9f8-b11b-454c-b1aa-f859e56d4389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" Oct 30 01:30:03.592850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:03.592830 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerStarted","Data":"c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66"} Oct 30 01:30:04.602052 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.602019 5548 scope.go:122] "RemoveContainer" containerID="48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51" Oct 30 01:30:04.602528 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:04.602362 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c_odh-ai-gateway-infra(8988d9f8-b11b-454c-b1aa-f859e56d4389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" Oct 30 01:30:04.603824 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.603800 5548 generic.go:356] "Generic (PLEG): container finished" podID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerID="c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66" exitCode=60 Oct 30 01:30:04.603965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.603951 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerDied","Data":"c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66"} Oct 30 01:30:04.604012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.603996 5548 scope.go:122] "RemoveContainer" containerID="d3eb8653888637088d7d8aa9750f8ea01206ca2644511d531ae0c27d8d0fb9d6" Oct 30 01:30:04.604275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.604257 5548 scope.go:122] "RemoveContainer" containerID="c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66" Oct 30 01:30:04.604554 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:04.604529 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c_odh-ai-gateway-infra(1edf8627-750a-48a8-a30d-d8106888e2d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" Oct 30 01:30:04.879910 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.879804 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" containerID="cri-o://cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d" gracePeriod=45 Oct 30 01:30:04.901352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:04.901316 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" containerID="cri-o://268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15" gracePeriod=45 Oct 30 01:30:05.205955 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.205928 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:30:05.234822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.234798 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.292875 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.292937 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.292998 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293048 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293100 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\" (UniqueName: \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293122 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293139 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\") pod \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\" (UID: \"9d360b3e-9afb-4f29-abd4-eef10ce7a89c\") " Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293768 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.293939 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.294721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.294003 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.298545 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.297919 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "kube-api-access-qjc2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.298800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.298780 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:05.299938 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.299913 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm" pod "9d360b3e-9afb-4f29-abd4-eef10ce7a89c" (UID: "9d360b3e-9afb-4f29-abd4-eef10ce7a89c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.395090 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395057 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395148 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395196 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395275 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\" (UniqueName: \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395362 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395384 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\") pod \"d740d6f4-0229-45e3-8b13-ddd190c044bc\" (UID: \"d740d6f4-0229-45e3-8b13-ddd190c044bc\") " Oct 30 01:30:05.395816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395763 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.395816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395789 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.395816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395801 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.395816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395816 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjc2j\" (UniqueName: \"kubernetes.io/projected/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-kube-api-access-qjc2j\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.396153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395830 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d360b3e-9afb-4f29-abd4-eef10ce7a89c-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.396153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.395843 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.396153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.396040 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.396432 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.396377 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.398493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.398469 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:05.398595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.398522 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "kube-api-access-zp4wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.398704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.398687 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm" pod "d740d6f4-0229-45e3-8b13-ddd190c044bc" (UID: "d740d6f4-0229-45e3-8b13-ddd190c044bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.496829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496786 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4wm\" (UniqueName: \"kubernetes.io/projected/d740d6f4-0229-45e3-8b13-ddd190c044bc-kube-api-access-zp4wm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.496829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496822 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.496829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496831 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.496829 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496842 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d740d6f4-0229-45e3-8b13-ddd190c044bc-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.497281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496850 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.497281 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.496878 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d740d6f4-0229-45e3-8b13-ddd190c044bc-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.512828 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.512781 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" containerID="cri-o://6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5" gracePeriod=45 Oct 30 01:30:05.612342 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.612306 5548 generic.go:356] "Generic (PLEG): container finished" podID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerID="cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d" exitCode=0 Oct 30 01:30:05.612800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.612358 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" Oct 30 01:30:05.612800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.612368 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" event={"ID":"9d360b3e-9afb-4f29-abd4-eef10ce7a89c","Type":"ContainerDied","Data":"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d"} Oct 30 01:30:05.612800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.612410 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr" event={"ID":"9d360b3e-9afb-4f29-abd4-eef10ce7a89c","Type":"ContainerDied","Data":"0a6cb1be2927cc5fd4d3432809dacdcdbae4b7d466b446df30df2f9abb28f73a"} Oct 30 01:30:05.612800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.612433 5548 scope.go:122] "RemoveContainer" containerID="cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d" Oct 30 01:30:05.614750 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.614729 5548 scope.go:122] "RemoveContainer" containerID="c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66" Oct 30 01:30:05.615144 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:05.615113 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c_odh-ai-gateway-infra(1edf8627-750a-48a8-a30d-d8106888e2d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" Oct 30 01:30:05.615874 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.615836 5548 generic.go:356] "Generic (PLEG): container finished" podID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerID="268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15" exitCode=0 Oct 30 01:30:05.615947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.615883 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" Oct 30 01:30:05.615947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.615893 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" event={"ID":"d740d6f4-0229-45e3-8b13-ddd190c044bc","Type":"ContainerDied","Data":"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15"} Oct 30 01:30:05.615947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.615930 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97" event={"ID":"d740d6f4-0229-45e3-8b13-ddd190c044bc","Type":"ContainerDied","Data":"b874004160de672aa84d7dfc1ef89745240a5311804537b7d3e91c8e3a49b1cc"} Oct 30 01:30:05.628088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.628064 5548 scope.go:122] "RemoveContainer" containerID="cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d" Oct 30 01:30:05.628369 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:05.628349 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d\": container with ID starting with cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d not found: ID does not exist" containerID="cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d" Oct 30 01:30:05.628447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.628377 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d"} err="failed to get container status \"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d\": rpc error: code = NotFound desc = could not find container \"cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d\": container with ID starting with cdfc79e377e5efe9d7a53f14a704825600d81bb78f8a4d78d9667c9129d23b8d not found: ID does not exist" Oct 30 01:30:05.628447 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.628392 5548 scope.go:122] "RemoveContainer" containerID="268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15" Oct 30 01:30:05.639964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.639940 5548 scope.go:122] "RemoveContainer" containerID="268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15" Oct 30 01:30:05.640316 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:05.640287 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15\": container with ID starting with 268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15 not found: ID does not exist" containerID="268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15" Oct 30 01:30:05.640412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.640325 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15"} err="failed to get container status \"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15\": rpc error: code = NotFound desc = could not find container \"268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15\": container with ID starting with 268360d699002e03f19660f0910e052f91200fa93859e56c2d77d32aee21bf15 not found: ID does not exist" Oct 30 01:30:05.704483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.704422 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 01:30:05.710826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.710796 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/premium-w4-d827b7-kserve-7dc964f56b-rpn97"] Oct 30 01:30:05.744055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.744025 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 01:30:05.748524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.748494 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d827b7/distinct-2-w4-d827b7-kserve-74669b78c8-fwsdr"] Oct 30 01:30:05.764463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.764441 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:30:05.900961 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.900924 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\" (UniqueName: \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.901148 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901044 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.901226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901155 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.901665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901374 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.901665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901413 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.901665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901454 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.901665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901542 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.902081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901651 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\" (UniqueName: \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\") pod \"48654eb2-d22d-444f-b49a-e2f01c69d499\" (UID: \"48654eb2-d22d-444f-b49a-e2f01c69d499\") " Oct 30 01:30:05.902081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.901938 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.902544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.902519 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.902631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.902549 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.902631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.902564 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.904497 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.904077 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "kube-api-access-l4p7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.905010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.904981 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.905162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:05.905136 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs" pod "48654eb2-d22d-444f-b49a-e2f01c69d499" (UID: "48654eb2-d22d-444f-b49a-e2f01c69d499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:06.003595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.003508 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4p7q\" (UniqueName: \"kubernetes.io/projected/48654eb2-d22d-444f-b49a-e2f01c69d499-kube-api-access-l4p7q\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:06.003595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.003547 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48654eb2-d22d-444f-b49a-e2f01c69d499-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:06.003595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.003560 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48654eb2-d22d-444f-b49a-e2f01c69d499-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:06.604635 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.604606 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz"] Oct 30 01:30:06.605197 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605184 5548 cpu_manager.go:414] "containerMap: removing container" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" Oct 30 01:30:06.605258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605200 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" Oct 30 01:30:06.605258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605216 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" Oct 30 01:30:06.605258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605222 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" Oct 30 01:30:06.605258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605233 5548 cpu_manager.go:414] "containerMap: removing container" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" Oct 30 01:30:06.605258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605239 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" Oct 30 01:30:06.605411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605317 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" containerName="main" Oct 30 01:30:06.605411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605325 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" containerName="main" Oct 30 01:30:06.605411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.605339 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" Oct 30 01:30:06.608632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.608549 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.626694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.626650 5548 generic.go:356] "Generic (PLEG): container finished" podID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerID="6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5" exitCode=0 Oct 30 01:30:06.627160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.626699 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" Oct 30 01:30:06.627160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.626819 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" event={"ID":"48654eb2-d22d-444f-b49a-e2f01c69d499","Type":"ContainerDied","Data":"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5"} Oct 30 01:30:06.627160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.626848 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" event={"ID":"48654eb2-d22d-444f-b49a-e2f01c69d499","Type":"ContainerDied","Data":"a1e914426e4dbd21714ce996272ff80294d0d068afaaf6e06cabda65f2d05929"} Oct 30 01:30:06.627160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.626889 5548 scope.go:122] "RemoveContainer" containerID="6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5" Oct 30 01:30:06.638058 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.637334 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:06.646577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.645065 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz"] Oct 30 01:30:06.653336 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.653309 5548 scope.go:122] "RemoveContainer" containerID="6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5" Oct 30 01:30:06.660260 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:06.660229 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5\": container with ID starting with 6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5 not found: ID does not exist" containerID="6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5" Oct 30 01:30:06.660358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.660268 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5"} err="failed to get container status \"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5\": rpc error: code = NotFound desc = could not find container \"6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5\": container with ID starting with 6d44a024ba5055ff5d276c6cffb48d238de0e855745df8a0e06c661157a05fc5 not found: ID does not exist" Oct 30 01:30:06.703189 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.703159 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 01:30:06.713392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.713366 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.713515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.713419 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd5c\" (UniqueName: \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.713515 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.713439 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.714930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.714906 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:30:06.721623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.721601 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/simulator-w1-c104b8-kserve-6b4b66784b-dw8pj"] Oct 30 01:30:06.814527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.814495 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.814673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.814550 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd5c\" (UniqueName: \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.814673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.814580 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.815172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.815141 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.817233 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.817210 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.832538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.832508 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd5c\" (UniqueName: \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\") pod \"maas-api-revoke-sub-600aaa78-cqbfz\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:06.924943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:06.924917 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:07.239289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.239260 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz"] Oct 30 01:30:07.240494 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:30:07.240466 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897bdf22_f0c5_48b7_9727_713117e85343.slice/crio-7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b WatchSource:0}: Error finding container 7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b: Status 404 returned error can't find the container with id 7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b Oct 30 01:30:07.264151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.264125 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48654eb2-d22d-444f-b49a-e2f01c69d499" path="/var/lib/kubelet/pods/48654eb2-d22d-444f-b49a-e2f01c69d499/volumes" Oct 30 01:30:07.264631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.264616 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d360b3e-9afb-4f29-abd4-eef10ce7a89c" path="/var/lib/kubelet/pods/9d360b3e-9afb-4f29-abd4-eef10ce7a89c/volumes" Oct 30 01:30:07.265132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.265112 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d740d6f4-0229-45e3-8b13-ddd190c044bc" path="/var/lib/kubelet/pods/d740d6f4-0229-45e3-8b13-ddd190c044bc/volumes" Oct 30 01:30:07.637351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.637312 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" event={"ID":"897bdf22-f0c5-48b7-9727-713117e85343","Type":"ContainerStarted","Data":"1118fecabe981b72d630cb646c00accfd055fcf29ead3d8cde7cab037c7136c5"} Oct 30 01:30:07.637808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.637385 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" event={"ID":"897bdf22-f0c5-48b7-9727-713117e85343","Type":"ContainerStarted","Data":"7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b"} Oct 30 01:30:07.680124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.680073 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" podStartSLOduration=1.680056761 podStartE2EDuration="1.680056761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:07.676980386 +0000 UTC m=+2798.519540991" watchObservedRunningTime="2026-10-30 01:30:07.680056761 +0000 UTC m=+2798.522617355" Oct 30 01:30:07.978188 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.978149 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:30:07.978443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:07.978418 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" containerID="cri-o://20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790" gracePeriod=30 Oct 30 01:30:08.647137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.647098 5548 generic.go:356] "Generic (PLEG): container finished" podID="897bdf22-f0c5-48b7-9727-713117e85343" containerID="1118fecabe981b72d630cb646c00accfd055fcf29ead3d8cde7cab037c7136c5" exitCode=0 Oct 30 01:30:08.647628 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.647189 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" event={"ID":"897bdf22-f0c5-48b7-9727-713117e85343","Type":"ContainerDied","Data":"1118fecabe981b72d630cb646c00accfd055fcf29ead3d8cde7cab037c7136c5"} Oct 30 01:30:08.701277 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.701241 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5"] Oct 30 01:30:08.705271 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.705252 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.748448 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.748417 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.748574 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.748476 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.748714 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.748688 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89qr\" (UniqueName: \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.811348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.811311 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5"] Oct 30 01:30:08.850174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.849826 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.850174 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.849963 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.850444 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.850241 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89qr\" (UniqueName: \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.850986 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.850954 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.853247 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.853223 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:08.877381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:08.877355 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89qr\" (UniqueName: \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\") pod \"maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:09.018319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.018241 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:09.332359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.332334 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5"] Oct 30 01:30:09.334670 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:30:09.334642 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392e1d3c_dd4f_48ec_8a69_6ca762010d1a.slice/crio-8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9 WatchSource:0}: Error finding container 8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9: Status 404 returned error can't find the container with id 8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9 Oct 30 01:30:09.655261 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.655226 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" event={"ID":"392e1d3c-dd4f-48ec-8a69-6ca762010d1a","Type":"ContainerStarted","Data":"8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9"} Oct 30 01:30:09.810381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.810354 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:09.863351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.863285 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\" (UniqueName: \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\") pod \"897bdf22-f0c5-48b7-9727-713117e85343\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " Oct 30 01:30:09.863351 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.863342 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\") pod \"897bdf22-f0c5-48b7-9727-713117e85343\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " Oct 30 01:30:09.863540 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.863432 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\") pod \"897bdf22-f0c5-48b7-9727-713117e85343\" (UID: \"897bdf22-f0c5-48b7-9727-713117e85343\") " Oct 30 01:30:09.863795 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.863766 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca" pod "897bdf22-f0c5-48b7-9727-713117e85343" (UID: "897bdf22-f0c5-48b7-9727-713117e85343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:09.863997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.863978 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:09.866297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.866266 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c" pod "897bdf22-f0c5-48b7-9727-713117e85343" (UID: "897bdf22-f0c5-48b7-9727-713117e85343"). InnerVolumeSpecName "kube-api-access-gvd5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:09.867471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.867431 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca" pod "897bdf22-f0c5-48b7-9727-713117e85343" (UID: "897bdf22-f0c5-48b7-9727-713117e85343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:09.965120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.965092 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd5c\" (UniqueName: \"kubernetes.io/projected/897bdf22-f0c5-48b7-9727-713117e85343-kube-api-access-gvd5c\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:09.965120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:09.965119 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/897bdf22-f0c5-48b7-9727-713117e85343-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:10.663760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.663723 5548 generic.go:356] "Generic (PLEG): container finished" podID="392e1d3c-dd4f-48ec-8a69-6ca762010d1a" containerID="3bb6fd546f8d3b2a6451c60da784aa0ff7867d237c96e2dd6fe468fa2e9d7f07" exitCode=0 Oct 30 01:30:10.664242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.663814 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" event={"ID":"392e1d3c-dd4f-48ec-8a69-6ca762010d1a","Type":"ContainerDied","Data":"3bb6fd546f8d3b2a6451c60da784aa0ff7867d237c96e2dd6fe468fa2e9d7f07"} Oct 30 01:30:10.665543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.665523 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" Oct 30 01:30:10.665672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.665541 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz" event={"ID":"897bdf22-f0c5-48b7-9727-713117e85343","Type":"ContainerDied","Data":"7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b"} Oct 30 01:30:10.665672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.665575 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb63c4987db2258c6f537d6d67feaf2774cd7af3bd316bb8450f7ce1f7df02b" Oct 30 01:30:10.679379 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.679359 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:30:10.679538 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.679518 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" podUID="fecfccee-49fb-43d2-8382-d9f4880f78db" containerName="maas-api" containerID="cri-o://39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997" gracePeriod=30 Oct 30 01:30:10.777706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.777666 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:10.859273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.856756 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:30:10.859273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.857034 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" podUID="083f1262-dc88-489a-b84d-f0d76c32ab4f" containerName="payload-processing" containerID="cri-o://11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a" gracePeriod=30 Oct 30 01:30:10.882089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.882063 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:30:10.883116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:10.882292 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" podUID="c6504af3-1d50-4b0f-9e02-043033355293" containerName="payload-pre-processing" containerID="cri-o://a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947" gracePeriod=30 Oct 30 01:30:11.038027 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.037681 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:11.075283 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.075181 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:30:11.084546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.083312 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\" (UniqueName: \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\") pod \"1edf8627-750a-48a8-a30d-d8106888e2d1\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " Oct 30 01:30:11.084546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.083591 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\") pod \"1edf8627-750a-48a8-a30d-d8106888e2d1\" (UID: \"1edf8627-750a-48a8-a30d-d8106888e2d1\") " Oct 30 01:30:11.084769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.084631 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca" pod "1edf8627-750a-48a8-a30d-d8106888e2d1" (UID: "1edf8627-750a-48a8-a30d-d8106888e2d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.087904 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.087719 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq" pod "1edf8627-750a-48a8-a30d-d8106888e2d1" (UID: "1edf8627-750a-48a8-a30d-d8106888e2d1"). InnerVolumeSpecName "kube-api-access-6t7lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.185557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.185526 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\") pod \"fecfccee-49fb-43d2-8382-d9f4880f78db\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " Oct 30 01:30:11.185705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.185612 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\" (UniqueName: \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\") pod \"fecfccee-49fb-43d2-8382-d9f4880f78db\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " Oct 30 01:30:11.185705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.185646 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\") pod \"fecfccee-49fb-43d2-8382-d9f4880f78db\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " Oct 30 01:30:11.185705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.185694 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\") pod \"fecfccee-49fb-43d2-8382-d9f4880f78db\" (UID: \"fecfccee-49fb-43d2-8382-d9f4880f78db\") " Oct 30 01:30:11.186162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.186144 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edf8627-750a-48a8-a30d-d8106888e2d1-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.186329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.186166 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7lq\" (UniqueName: \"kubernetes.io/projected/1edf8627-750a-48a8-a30d-d8106888e2d1-kube-api-access-6t7lq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.187237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.187200 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca" pod "fecfccee-49fb-43d2-8382-d9f4880f78db" (UID: "fecfccee-49fb-43d2-8382-d9f4880f78db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.189225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.189197 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs" pod "fecfccee-49fb-43d2-8382-d9f4880f78db" (UID: "fecfccee-49fb-43d2-8382-d9f4880f78db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.193736 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.193712 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:30:11.194033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.194008 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd" pod "fecfccee-49fb-43d2-8382-d9f4880f78db" (UID: "fecfccee-49fb-43d2-8382-d9f4880f78db"). InnerVolumeSpecName "kube-api-access-hqpjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.196024 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.195990 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls" pod "fecfccee-49fb-43d2-8382-d9f4880f78db" (UID: "fecfccee-49fb-43d2-8382-d9f4880f78db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.357938 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\") pod \"083f1262-dc88-489a-b84d-f0d76c32ab4f\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.358049 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\" (UniqueName: \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\") pod \"083f1262-dc88-489a-b84d-f0d76c32ab4f\" (UID: \"083f1262-dc88-489a-b84d-f0d76c32ab4f\") " Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.358542 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.358565 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpjd\" (UniqueName: \"kubernetes.io/projected/fecfccee-49fb-43d2-8382-d9f4880f78db-kube-api-access-hqpjd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.358579 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecfccee-49fb-43d2-8382-d9f4880f78db-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.358678 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fecfccee-49fb-43d2-8382-d9f4880f78db-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.360111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.359020 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume" pod "083f1262-dc88-489a-b84d-f0d76c32ab4f" (UID: "083f1262-dc88-489a-b84d-f0d76c32ab4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.367989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.366801 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x" pod "083f1262-dc88-489a-b84d-f0d76c32ab4f" (UID: "083f1262-dc88-489a-b84d-f0d76c32ab4f"). InnerVolumeSpecName "kube-api-access-h665x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.408490 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.408465 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:30:11.459769 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.459599 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\" (UniqueName: \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\") pod \"c6504af3-1d50-4b0f-9e02-043033355293\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " Oct 30 01:30:11.460000 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.459848 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\") pod \"c6504af3-1d50-4b0f-9e02-043033355293\" (UID: \"c6504af3-1d50-4b0f-9e02-043033355293\") " Oct 30 01:30:11.460273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.460234 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume" pod "c6504af3-1d50-4b0f-9e02-043033355293" (UID: "c6504af3-1d50-4b0f-9e02-043033355293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.460407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.460368 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6504af3-1d50-4b0f-9e02-043033355293-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.460407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.460389 5548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/083f1262-dc88-489a-b84d-f0d76c32ab4f-plugins-config-volume\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.460407 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.460402 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h665x\" (UniqueName: \"kubernetes.io/projected/083f1262-dc88-489a-b84d-f0d76c32ab4f-kube-api-access-h665x\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.462672 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.462647 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8" pod "c6504af3-1d50-4b0f-9e02-043033355293" (UID: "c6504af3-1d50-4b0f-9e02-043033355293"). InnerVolumeSpecName "kube-api-access-xwwn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.561532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.561494 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwn8\" (UniqueName: \"kubernetes.io/projected/c6504af3-1d50-4b0f-9e02-043033355293-kube-api-access-xwwn8\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.688198 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.688166 5548 generic.go:356] "Generic (PLEG): container finished" podID="c6504af3-1d50-4b0f-9e02-043033355293" containerID="a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947" exitCode=0 Oct 30 01:30:11.688702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.688216 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" Oct 30 01:30:11.688702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.688244 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" event={"ID":"c6504af3-1d50-4b0f-9e02-043033355293","Type":"ContainerDied","Data":"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947"} Oct 30 01:30:11.688702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.688323 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld" event={"ID":"c6504af3-1d50-4b0f-9e02-043033355293","Type":"ContainerDied","Data":"2556a218d8fd9989add87aa48c55540df2fb56a5a90134e273a4d3db130a134e"} Oct 30 01:30:11.688702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.688371 5548 scope.go:122] "RemoveContainer" containerID="a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947" Oct 30 01:30:11.691304 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.691283 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" Oct 30 01:30:11.691551 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.691288 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c" event={"ID":"1edf8627-750a-48a8-a30d-d8106888e2d1","Type":"ContainerDied","Data":"ee01d9697efd18f657e326d5ea561766878e9c542526ffeb49c8b147200b08bf"} Oct 30 01:30:11.693135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.693109 5548 generic.go:356] "Generic (PLEG): container finished" podID="fecfccee-49fb-43d2-8382-d9f4880f78db" containerID="39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997" exitCode=0 Oct 30 01:30:11.693239 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.693221 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" event={"ID":"fecfccee-49fb-43d2-8382-d9f4880f78db","Type":"ContainerDied","Data":"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997"} Oct 30 01:30:11.693316 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.693265 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" event={"ID":"fecfccee-49fb-43d2-8382-d9f4880f78db","Type":"ContainerDied","Data":"64838f5927b21b7d6eda783d7a409b130fdc48bf79b73a0cb0d33c9186108bd6"} Oct 30 01:30:11.693386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.693335 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx" Oct 30 01:30:11.696222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.696193 5548 generic.go:356] "Generic (PLEG): container finished" podID="083f1262-dc88-489a-b84d-f0d76c32ab4f" containerID="11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a" exitCode=0 Oct 30 01:30:11.696482 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.696368 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" event={"ID":"083f1262-dc88-489a-b84d-f0d76c32ab4f","Type":"ContainerDied","Data":"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a"} Oct 30 01:30:11.696482 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.696418 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" event={"ID":"083f1262-dc88-489a-b84d-f0d76c32ab4f","Type":"ContainerDied","Data":"4adc0a216f479721f3b7069005b22e4bb778e04dca38c11fce6b13cd49218b19"} Oct 30 01:30:11.696616 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.696483 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd" Oct 30 01:30:11.703781 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.703764 5548 scope.go:122] "RemoveContainer" containerID="a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947" Oct 30 01:30:11.704094 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:11.704074 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947\": container with ID starting with a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947 not found: ID does not exist" containerID="a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947" Oct 30 01:30:11.704180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.704102 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947"} err="failed to get container status \"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947\": rpc error: code = NotFound desc = could not find container \"a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947\": container with ID starting with a2f2bb4d03fac8a918ddbca8c221f5fed5247662617c1db20fbeefcc98539947 not found: ID does not exist" Oct 30 01:30:11.704180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.704118 5548 scope.go:122] "RemoveContainer" containerID="c9669d8dd4737f6e1c66a7ba210f1bc614b20dc443e31b9e90d4797cbbc19d66" Oct 30 01:30:11.713744 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.713722 5548 scope.go:122] "RemoveContainer" containerID="39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997" Oct 30 01:30:11.723813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.723792 5548 scope.go:122] "RemoveContainer" containerID="39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997" Oct 30 01:30:11.724212 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:11.724193 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997\": container with ID starting with 39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997 not found: ID does not exist" containerID="39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997" Oct 30 01:30:11.724272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.724225 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997"} err="failed to get container status \"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997\": rpc error: code = NotFound desc = could not find container \"39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997\": container with ID starting with 39a9e811d0c1ab73eb452abd0cc95c4864d798eab16b88a16cd081c32b48e997 not found: ID does not exist" Oct 30 01:30:11.724272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.724246 5548 scope.go:122] "RemoveContainer" containerID="11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a" Oct 30 01:30:11.735615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.735588 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:30:11.736592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.736532 5548 scope.go:122] "RemoveContainer" containerID="11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a" Oct 30 01:30:11.737646 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:11.737615 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a\": container with ID starting with 11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a not found: ID does not exist" containerID="11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a" Oct 30 01:30:11.737748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.737649 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a"} err="failed to get container status \"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a\": rpc error: code = NotFound desc = could not find container \"11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a\": container with ID starting with 11af464769448352cf82e65e503490d19228679595592c1bb420093850ca299a not found: ID does not exist" Oct 30 01:30:11.740688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.740661 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-368bbb-6d7bffb559-s86ld"] Oct 30 01:30:11.774926 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.774893 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:30:11.786878 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.786842 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-368bbb-55769f5bb4-8bhjd"] Oct 30 01:30:11.813581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.813546 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:11.816967 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.816940 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-368bbb-29888730-2kt4c"] Oct 30 01:30:11.838274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.838248 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:11.851974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.851943 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:30:11.854558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.854540 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-368bbb-5dd96d6fb6-g6dhx"] Oct 30 01:30:11.865668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.865642 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\") pod \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " Oct 30 01:30:11.865668 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.865677 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\" (UniqueName: \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\") pod \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " Oct 30 01:30:11.865952 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.865820 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\") pod \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\" (UID: \"392e1d3c-dd4f-48ec-8a69-6ca762010d1a\") " Oct 30 01:30:11.866131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.866107 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca" pod "392e1d3c-dd4f-48ec-8a69-6ca762010d1a" (UID: "392e1d3c-dd4f-48ec-8a69-6ca762010d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.866392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.866373 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.868064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.868044 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca" pod "392e1d3c-dd4f-48ec-8a69-6ca762010d1a" (UID: "392e1d3c-dd4f-48ec-8a69-6ca762010d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.868150 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.868135 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr" pod "392e1d3c-dd4f-48ec-8a69-6ca762010d1a" (UID: "392e1d3c-dd4f-48ec-8a69-6ca762010d1a"). InnerVolumeSpecName "kube-api-access-b89qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.967208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.967131 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89qr\" (UniqueName: \"kubernetes.io/projected/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-kube-api-access-b89qr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.967208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:11.967160 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392e1d3c-dd4f-48ec-8a69-6ca762010d1a-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:12.705307 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:12.705275 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" Oct 30 01:30:12.705728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:12.705280 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5" event={"ID":"392e1d3c-dd4f-48ec-8a69-6ca762010d1a","Type":"ContainerDied","Data":"8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9"} Oct 30 01:30:12.705728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:12.705400 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d8107465d0bd696dbd14ca502d7bde995a7a2bb2bd47ff6f7a5c7d8d5d094d9" Oct 30 01:30:13.268020 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.267933 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083f1262-dc88-489a-b84d-f0d76c32ab4f" path="/var/lib/kubelet/pods/083f1262-dc88-489a-b84d-f0d76c32ab4f/volumes" Oct 30 01:30:13.268425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.268407 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" path="/var/lib/kubelet/pods/1edf8627-750a-48a8-a30d-d8106888e2d1/volumes" Oct 30 01:30:13.268839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.268822 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6504af3-1d50-4b0f-9e02-043033355293" path="/var/lib/kubelet/pods/c6504af3-1d50-4b0f-9e02-043033355293/volumes" Oct 30 01:30:13.269549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.269532 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecfccee-49fb-43d2-8382-d9f4880f78db" path="/var/lib/kubelet/pods/fecfccee-49fb-43d2-8382-d9f4880f78db/volumes" Oct 30 01:30:13.322752 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.322727 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:30:13.418475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418439 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418514 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418551 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418627 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418696 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418675 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418730 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:13.418957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418744 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418844 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.418957 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.418945 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.419176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419000 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\") pod \"d469e1b4-394b-4288-988e-7d5725f3a218\" (UID: \"d469e1b4-394b-4288-988e-7d5725f3a218\") " Oct 30 01:30:13.419491 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419428 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.419688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419668 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:13.419765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419709 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:13.419893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419872 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:13.420002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.419976 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:13.422694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.421846 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:13.422694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.421933 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:13.422694 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.422093 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "kube-api-access-7rndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:13.425048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.425019 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token" pod "d469e1b4-394b-4288-988e-7d5725f3a218" (UID: "d469e1b4-394b-4288-988e-7d5725f3a218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520350 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520380 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rndr\" (UniqueName: \"kubernetes.io/projected/d469e1b4-394b-4288-988e-7d5725f3a218-kube-api-access-7rndr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520391 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d469e1b4-394b-4288-988e-7d5725f3a218-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520399 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520408 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520415 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520423 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d469e1b4-394b-4288-988e-7d5725f3a218-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.520429 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.520430 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d469e1b4-394b-4288-988e-7d5725f3a218-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.718133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.718100 5548 generic.go:356] "Generic (PLEG): container finished" podID="d469e1b4-394b-4288-988e-7d5725f3a218" containerID="20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790" exitCode=0 Oct 30 01:30:13.718637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.718151 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" Oct 30 01:30:13.718637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.718194 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerDied","Data":"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790"} Oct 30 01:30:13.718637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.718251 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd" event={"ID":"d469e1b4-394b-4288-988e-7d5725f3a218","Type":"ContainerDied","Data":"5c3350177d3121d8abdfe7cfa5b6d54321c2c6e394601a3c01b1f30fe058ff80"} Oct 30 01:30:13.718637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.718275 5548 scope.go:122] "RemoveContainer" containerID="20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790" Oct 30 01:30:13.730225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.730205 5548 scope.go:122] "RemoveContainer" containerID="39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453" Oct 30 01:30:13.739731 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.739711 5548 scope.go:122] "RemoveContainer" containerID="20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790" Oct 30 01:30:13.740014 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:13.739996 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790\": container with ID starting with 20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790 not found: ID does not exist" containerID="20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790" Oct 30 01:30:13.740080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.740023 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790"} err="failed to get container status \"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790\": rpc error: code = NotFound desc = could not find container \"20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790\": container with ID starting with 20c0c2f9118f83dd6380cd44730a01efdde69bf03027aa668afe4ea0c222b790 not found: ID does not exist" Oct 30 01:30:13.740080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.740039 5548 scope.go:122] "RemoveContainer" containerID="39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453" Oct 30 01:30:13.740315 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:13.740300 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453\": container with ID starting with 39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453 not found: ID does not exist" containerID="39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453" Oct 30 01:30:13.740353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.740321 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453"} err="failed to get container status \"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453\": rpc error: code = NotFound desc = could not find container \"39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453\": container with ID starting with 39c0f4481f442f84a67095a217289c37e579e478511130938cbcced852ead453 not found: ID does not exist" Oct 30 01:30:13.770798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.770729 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:30:13.781038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:13.781014 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-dbmrd"] Oct 30 01:30:15.264807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:15.264773 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" path="/var/lib/kubelet/pods/d469e1b4-394b-4288-988e-7d5725f3a218/volumes" Oct 30 01:30:18.257167 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:18.257136 5548 scope.go:122] "RemoveContainer" containerID="48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51" Oct 30 01:30:18.775947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:18.775907 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerStarted","Data":"54eeba68c630c653f184d6702320416252758f6bf9497362377ec1ba351a7ad6"} Oct 30 01:30:19.784483 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:19.784443 5548 generic.go:356] "Generic (PLEG): container finished" podID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerID="54eeba68c630c653f184d6702320416252758f6bf9497362377ec1ba351a7ad6" exitCode=60 Oct 30 01:30:19.784994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:19.784514 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerDied","Data":"54eeba68c630c653f184d6702320416252758f6bf9497362377ec1ba351a7ad6"} Oct 30 01:30:19.784994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:19.784602 5548 scope.go:122] "RemoveContainer" containerID="48f4ea19eeedef7408a49c50599d3f9bbbf7018ae08dd5a8b8ad100b76823f51" Oct 30 01:30:19.784994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:19.784982 5548 scope.go:122] "RemoveContainer" containerID="54eeba68c630c653f184d6702320416252758f6bf9497362377ec1ba351a7ad6" Oct 30 01:30:19.785326 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:19.785305 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c_odh-ai-gateway-infra(8988d9f8-b11b-454c-b1aa-f859e56d4389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" Oct 30 01:30:19.848115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:19.848084 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:20.950084 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:20.950056 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:21.021293 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.021266 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\" (UniqueName: \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\") pod \"8988d9f8-b11b-454c-b1aa-f859e56d4389\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " Oct 30 01:30:21.021492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.021414 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\") pod \"8988d9f8-b11b-454c-b1aa-f859e56d4389\" (UID: \"8988d9f8-b11b-454c-b1aa-f859e56d4389\") " Oct 30 01:30:21.021819 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.021787 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca" pod "8988d9f8-b11b-454c-b1aa-f859e56d4389" (UID: "8988d9f8-b11b-454c-b1aa-f859e56d4389"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:21.024039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.024001 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6" pod "8988d9f8-b11b-454c-b1aa-f859e56d4389" (UID: "8988d9f8-b11b-454c-b1aa-f859e56d4389"). InnerVolumeSpecName "kube-api-access-m7ps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:21.123816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.123723 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ps6\" (UniqueName: \"kubernetes.io/projected/8988d9f8-b11b-454c-b1aa-f859e56d4389-kube-api-access-m7ps6\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:21.123816 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.123761 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8988d9f8-b11b-454c-b1aa-f859e56d4389-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:21.800613 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.800582 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" Oct 30 01:30:21.800825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.800619 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c" event={"ID":"8988d9f8-b11b-454c-b1aa-f859e56d4389","Type":"ContainerDied","Data":"edc942cc1f38a4b66f1a5f6cd971b9e0209cdc802018a617ae8efa1856471601"} Oct 30 01:30:21.800825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.800680 5548 scope.go:122] "RemoveContainer" containerID="54eeba68c630c653f184d6702320416252758f6bf9497362377ec1ba351a7ad6" Oct 30 01:30:21.862840 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.862807 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:21.870169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:21.870143 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-82b647-29888730-xdz9c"] Oct 30 01:30:23.264311 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:23.264274 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" path="/var/lib/kubelet/pods/8988d9f8-b11b-454c-b1aa-f859e56d4389/volumes" Oct 30 01:30:25.429374 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:25.429338 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:30:39.066180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.066147 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:30:39.067152 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.066406 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" podUID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" containerName="maas-api" containerID="cri-o://d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20" gracePeriod=30 Oct 30 01:30:39.400435 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.400409 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:30:39.476050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.476019 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\") pod \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " Oct 30 01:30:39.476275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.476074 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\") pod \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " Oct 30 01:30:39.476275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.476180 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\" (UniqueName: \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\") pod \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " Oct 30 01:30:39.476275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.476220 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\") pod \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\" (UID: \"89a5a2ec-5a4a-48a9-8841-8123894ba61f\") " Oct 30 01:30:39.476753 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.476722 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca" pod "89a5a2ec-5a4a-48a9-8841-8123894ba61f" (UID: "89a5a2ec-5a4a-48a9-8841-8123894ba61f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:39.478477 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.478448 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs" pod "89a5a2ec-5a4a-48a9-8841-8123894ba61f" (UID: "89a5a2ec-5a4a-48a9-8841-8123894ba61f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:39.478599 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.478586 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7" pod "89a5a2ec-5a4a-48a9-8841-8123894ba61f" (UID: "89a5a2ec-5a4a-48a9-8841-8123894ba61f"). InnerVolumeSpecName "kube-api-access-rbwh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:39.478795 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.478778 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls" pod "89a5a2ec-5a4a-48a9-8841-8123894ba61f" (UID: "89a5a2ec-5a4a-48a9-8841-8123894ba61f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:39.578292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.578252 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:39.578292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.578294 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a5a2ec-5a4a-48a9-8841-8123894ba61f-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:39.578570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.578307 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbwh7\" (UniqueName: \"kubernetes.io/projected/89a5a2ec-5a4a-48a9-8841-8123894ba61f-kube-api-access-rbwh7\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:39.578570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.578321 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89a5a2ec-5a4a-48a9-8841-8123894ba61f-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:39.941614 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.941576 5548 generic.go:356] "Generic (PLEG): container finished" podID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" containerID="d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20" exitCode=0 Oct 30 01:30:39.941817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.941658 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" Oct 30 01:30:39.941817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.941671 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" event={"ID":"89a5a2ec-5a4a-48a9-8841-8123894ba61f","Type":"ContainerDied","Data":"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20"} Oct 30 01:30:39.942001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.941821 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx" event={"ID":"89a5a2ec-5a4a-48a9-8841-8123894ba61f","Type":"ContainerDied","Data":"42fca97c527552611c7757ab44929b2b323cce23b9eb8927eccbdcadcc38a2cc"} Oct 30 01:30:39.942001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.941874 5548 scope.go:122] "RemoveContainer" containerID="d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20" Oct 30 01:30:39.953308 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.953290 5548 scope.go:122] "RemoveContainer" containerID="d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20" Oct 30 01:30:39.953586 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:39.953569 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20\": container with ID starting with d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20 not found: ID does not exist" containerID="d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20" Oct 30 01:30:39.953637 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:39.953596 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20"} err="failed to get container status \"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20\": rpc error: code = NotFound desc = could not find container \"d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20\": container with ID starting with d2ba8c775be969d78375e24aa3bd570ac4ac88b9bd2ccba9176968030431cb20 not found: ID does not exist" Oct 30 01:30:40.002834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:40.002805 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:30:40.012103 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:40.012076 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-29bb56-7c74f9cf9-49slx"] Oct 30 01:30:41.265398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:41.265366 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" path="/var/lib/kubelet/pods/89a5a2ec-5a4a-48a9-8841-8123894ba61f/volumes" Oct 30 01:30:41.769636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:41.769600 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 01:30:41.770204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:41.770172 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" podUID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" containerName="maas-api" containerID="cri-o://2ba8b11b114ed69cd6d67ec7ddfb8288682a4f9fa5fbed085275588eea4648b6" gracePeriod=30 Oct 30 01:30:41.960711 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:41.960672 5548 generic.go:356] "Generic (PLEG): container finished" podID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" containerID="2ba8b11b114ed69cd6d67ec7ddfb8288682a4f9fa5fbed085275588eea4648b6" exitCode=0 Oct 30 01:30:41.960928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:41.960847 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" event={"ID":"d35f0cf9-21d8-4272-91fd-6acfc5235e42","Type":"ContainerDied","Data":"2ba8b11b114ed69cd6d67ec7ddfb8288682a4f9fa5fbed085275588eea4648b6"} Oct 30 01:30:42.072319 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.072231 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 01:30:42.196180 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.196134 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 01:30:42.196580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.196403 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" containerID="cri-o://3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79" gracePeriod=30 Oct 30 01:30:42.213169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.212021 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\") pod \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " Oct 30 01:30:42.213169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.212087 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\") pod \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " Oct 30 01:30:42.213169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.212287 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\") pod \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " Oct 30 01:30:42.213169 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.212325 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\" (UniqueName: \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\") pod \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\" (UID: \"d35f0cf9-21d8-4272-91fd-6acfc5235e42\") " Oct 30 01:30:42.214517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.214332 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca" pod "d35f0cf9-21d8-4272-91fd-6acfc5235e42" (UID: "d35f0cf9-21d8-4272-91fd-6acfc5235e42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:42.216519 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.216492 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv" pod "d35f0cf9-21d8-4272-91fd-6acfc5235e42" (UID: "d35f0cf9-21d8-4272-91fd-6acfc5235e42"). InnerVolumeSpecName "kube-api-access-8pmxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:42.217377 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.217346 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs" pod "d35f0cf9-21d8-4272-91fd-6acfc5235e42" (UID: "d35f0cf9-21d8-4272-91fd-6acfc5235e42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:42.217461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.217392 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls" pod "d35f0cf9-21d8-4272-91fd-6acfc5235e42" (UID: "d35f0cf9-21d8-4272-91fd-6acfc5235e42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:42.313723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.313684 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35f0cf9-21d8-4272-91fd-6acfc5235e42-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.313723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.313724 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmxv\" (UniqueName: \"kubernetes.io/projected/d35f0cf9-21d8-4272-91fd-6acfc5235e42-kube-api-access-8pmxv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.313723 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.313741 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.314398 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.313757 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d35f0cf9-21d8-4272-91fd-6acfc5235e42-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.585528 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.585492 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5"] Oct 30 01:30:42.588387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.588357 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-dca38d-fhtr5"] Oct 30 01:30:42.967243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.967210 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" Oct 30 01:30:42.967243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.967233 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7" event={"ID":"d35f0cf9-21d8-4272-91fd-6acfc5235e42","Type":"ContainerDied","Data":"051c95b6280d101c0d467cbf825f62cec055a7b7dac3bca58d819039a6ab507a"} Oct 30 01:30:42.967509 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:42.967295 5548 scope.go:122] "RemoveContainer" containerID="2ba8b11b114ed69cd6d67ec7ddfb8288682a4f9fa5fbed085275588eea4648b6" Oct 30 01:30:43.033124 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:43.033098 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 01:30:43.042887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:43.042849 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-dca38d-7d4bddf6d8-288p7"] Oct 30 01:30:43.265039 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:43.264965 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392e1d3c-dd4f-48ec-8a69-6ca762010d1a" path="/var/lib/kubelet/pods/392e1d3c-dd4f-48ec-8a69-6ca762010d1a/volumes" Oct 30 01:30:43.265409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:43.265395 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" path="/var/lib/kubelet/pods/d35f0cf9-21d8-4272-91fd-6acfc5235e42/volumes" Oct 30 01:30:47.125041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.125000 5548 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:30:47.125427 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.125062 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:30:47.482484 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.482453 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 01:30:47.588914 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.588879 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.588925 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.588977 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589041 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589082 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589112 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589189 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589242 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589270 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\") pod \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\" (UID: \"e0707bbd-1bd8-4708-89a1-f2f350fe50f3\") " Oct 30 01:30:47.589409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589314 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.589409 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589314 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.589674 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589568 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.589776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589760 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.589843 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589782 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.589843 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589797 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.590011 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.589969 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.590085 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.590033 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.591646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.591618 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:47.592104 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.592074 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:47.592275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.592253 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.592369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.592344 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf" pod "e0707bbd-1bd8-4708-89a1-f2f350fe50f3" (UID: "e0707bbd-1bd8-4708-89a1-f2f350fe50f3"). InnerVolumeSpecName "kube-api-access-r8hgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:47.690732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690702 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.690732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690730 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.690732 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690740 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.690973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690748 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.690973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690756 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8hgf\" (UniqueName: \"kubernetes.io/projected/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-kube-api-access-r8hgf\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.690973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:47.690766 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0707bbd-1bd8-4708-89a1-f2f350fe50f3-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:48.002626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.002592 5548 generic.go:356] "Generic (PLEG): container finished" podID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerID="3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79" exitCode=0 Oct 30 01:30:48.002813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.002640 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" Oct 30 01:30:48.002813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.002680 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" event={"ID":"e0707bbd-1bd8-4708-89a1-f2f350fe50f3","Type":"ContainerDied","Data":"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79"} Oct 30 01:30:48.002813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.002758 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx" event={"ID":"e0707bbd-1bd8-4708-89a1-f2f350fe50f3","Type":"ContainerDied","Data":"5c58b685757c5f33cca0213a99f8d64d55ca348c65e38f99948c284baf77437f"} Oct 30 01:30:48.002813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.002783 5548 scope.go:122] "RemoveContainer" containerID="3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79" Oct 30 01:30:48.013502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.013478 5548 scope.go:122] "RemoveContainer" containerID="3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79" Oct 30 01:30:48.013793 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:48.013774 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79\": container with ID starting with 3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79 not found: ID does not exist" containerID="3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79" Oct 30 01:30:48.013921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.013802 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79"} err="failed to get container status \"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79\": rpc error: code = NotFound desc = could not find container \"3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79\": container with ID starting with 3ad2ea8d3470d81ae31c9440419ae88a6c326716966b8683224c66f9101a2d79 not found: ID does not exist" Oct 30 01:30:48.053032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.053002 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 01:30:48.059652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.059630 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-sk2lx"] Oct 30 01:30:48.696583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.696549 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:30:48.697081 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:48.696746 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" containerID="cri-o://3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c" gracePeriod=30 Oct 30 01:30:49.264459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:49.264430 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" path="/var/lib/kubelet/pods/e0707bbd-1bd8-4708-89a1-f2f350fe50f3/volumes" Oct 30 01:30:49.872602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:49.872562 5548 patch_prober.go:29] interesting pod/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:30:49.873034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:49.872628 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:30:51.050800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:51.050766 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:30:51.051284 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:51.051159 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" containerID="cri-o://0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366" gracePeriod=30 Oct 30 01:30:53.995214 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:53.995192 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:30:54.042438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.042399 5548 generic.go:356] "Generic (PLEG): container finished" podID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerID="3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c" exitCode=0 Oct 30 01:30:54.042438 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.042444 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" Oct 30 01:30:54.042708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.042522 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerDied","Data":"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c"} Oct 30 01:30:54.042708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.042574 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr" event={"ID":"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e","Type":"ContainerDied","Data":"99c33470d155ae3c784c558da9580ac1f765ca444b34d36881a73a0c20a2acbb"} Oct 30 01:30:54.042708 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.042604 5548 scope.go:122] "RemoveContainer" containerID="3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c" Oct 30 01:30:54.056060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.055736 5548 scope.go:122] "RemoveContainer" containerID="c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124" Oct 30 01:30:54.086321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.086291 5548 scope.go:122] "RemoveContainer" containerID="3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c" Oct 30 01:30:54.086751 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:54.086719 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c\": container with ID starting with 3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c not found: ID does not exist" containerID="3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c" Oct 30 01:30:54.086978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.086762 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c"} err="failed to get container status \"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c\": rpc error: code = NotFound desc = could not find container \"3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c\": container with ID starting with 3c3ceb3528d9f1183649be61442d85d944641e11cdecfb5a7c6600bf10e8431c not found: ID does not exist" Oct 30 01:30:54.086978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.086785 5548 scope.go:122] "RemoveContainer" containerID="c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124" Oct 30 01:30:54.087145 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:54.087125 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124\": container with ID starting with c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124 not found: ID does not exist" containerID="c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124" Oct 30 01:30:54.087200 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.087154 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124"} err="failed to get container status \"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124\": rpc error: code = NotFound desc = could not find container \"c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124\": container with ID starting with c44d84867bb89a4ba4b7a4d2d112975173751b1cbbe6054120937ebbdaf35124 not found: ID does not exist" Oct 30 01:30:54.174670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174642 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174689 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174737 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174773 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174808 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174849 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174937 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174975 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174985 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:54.175274 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.174998 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\") pod \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\" (UID: \"ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e\") " Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.175643 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.175968 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.176064 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.176114 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.176159 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.178000 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.177844 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.178795 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "kube-api-access-qpzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:54.189595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.179268 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy" pod "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" (UID: "ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276802 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276835 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276844 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276851 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276893 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzxh\" (UniqueName: \"kubernetes.io/projected/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-kube-api-access-qpzxh\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276904 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276912 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.276928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.276920 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:54.420226 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.420190 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:30:54.447825 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:54.447797 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eaf753-openshift-default-f6fbcd6bc-tfkvr"] Oct 30 01:30:55.263671 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:55.263640 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" path="/var/lib/kubelet/pods/ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e/volumes" Oct 30 01:30:55.427587 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:55.427552 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:30:56.314581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.314559 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:30:56.500133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500101 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500225 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500266 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500305 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500371 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500446 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500482 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500519 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500539 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:56.500685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500584 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\") pod \"8865e1c8-27b0-4549-b818-2068e42c7cbb\" (UID: \"8865e1c8-27b0-4549-b818-2068e42c7cbb\") " Oct 30 01:30:56.500836 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500797 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:56.500836 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500801 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:56.500968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.500894 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:56.501096 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501075 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:56.501437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501399 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.501437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501425 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.501596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501438 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8865e1c8-27b0-4549-b818-2068e42c7cbb-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.501596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501452 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.501596 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.501465 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.502540 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.502518 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:56.502645 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.502616 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:56.503078 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.503058 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:56.503146 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.503076 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg" pod "8865e1c8-27b0-4549-b818-2068e42c7cbb" (UID: "8865e1c8-27b0-4549-b818-2068e42c7cbb"). InnerVolumeSpecName "kube-api-access-jxgdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:56.602047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.602015 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.602047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.602041 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxgdg\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-kube-api-access-jxgdg\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.602047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.602050 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:56.602262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:56.602059 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8865e1c8-27b0-4549-b818-2068e42c7cbb-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:30:57.063111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.063079 5548 generic.go:356] "Generic (PLEG): container finished" podID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerID="0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366" exitCode=0 Oct 30 01:30:57.063303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.063131 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" Oct 30 01:30:57.063303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.063159 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerDied","Data":"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366"} Oct 30 01:30:57.063303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.063211 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w" event={"ID":"8865e1c8-27b0-4549-b818-2068e42c7cbb","Type":"ContainerDied","Data":"8beb44ddbdd5f5d6b53b2d5b13e40a32e0c4c4a15a3769f5554cb7db6cc496d5"} Oct 30 01:30:57.063303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.063236 5548 scope.go:122] "RemoveContainer" containerID="0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366" Oct 30 01:30:57.073749 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.073723 5548 scope.go:122] "RemoveContainer" containerID="50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a" Oct 30 01:30:57.082737 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.082715 5548 scope.go:122] "RemoveContainer" containerID="0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366" Oct 30 01:30:57.082975 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:57.082956 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366\": container with ID starting with 0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366 not found: ID does not exist" containerID="0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366" Oct 30 01:30:57.083064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.082987 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366"} err="failed to get container status \"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366\": rpc error: code = NotFound desc = could not find container \"0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366\": container with ID starting with 0818f1d27173f846de05fe6f873f26088933ed81b53df64d6ef2bf60f1784366 not found: ID does not exist" Oct 30 01:30:57.083064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.083012 5548 scope.go:122] "RemoveContainer" containerID="50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a" Oct 30 01:30:57.083235 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:30:57.083218 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a\": container with ID starting with 50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a not found: ID does not exist" containerID="50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a" Oct 30 01:30:57.083272 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.083242 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a"} err="failed to get container status \"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a\": rpc error: code = NotFound desc = could not find container \"50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a\": container with ID starting with 50b59310cf29dc2e2f587f54e8ebbf91d8494fb8051a29a75192be7a5729c24a not found: ID does not exist" Oct 30 01:30:57.128440 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.128413 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:30:57.133127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.133104 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-29bb56-openshift-default-54bf4bb879-mkw7w"] Oct 30 01:30:57.264826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:30:57.264792 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" path="/var/lib/kubelet/pods/8865e1c8-27b0-4549-b818-2068e42c7cbb/volumes" Oct 30 01:31:25.428102 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:31:25.428063 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:31:33.669556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:33.669516 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:31:33.670023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:33.669823 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" podUID="eb4c6579-8d82-4b02-9680-0a5e57686732" containerName="maas-api" containerID="cri-o://be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44" gracePeriod=30 Oct 30 01:31:33.962698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:33.962603 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:31:34.024133 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.024098 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\") pod \"eb4c6579-8d82-4b02-9680-0a5e57686732\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " Oct 30 01:31:34.024348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.024155 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\" (UniqueName: \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\") pod \"eb4c6579-8d82-4b02-9680-0a5e57686732\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " Oct 30 01:31:34.024348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.024254 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\") pod \"eb4c6579-8d82-4b02-9680-0a5e57686732\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " Oct 30 01:31:34.024348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.024297 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\") pod \"eb4c6579-8d82-4b02-9680-0a5e57686732\" (UID: \"eb4c6579-8d82-4b02-9680-0a5e57686732\") " Oct 30 01:31:34.024730 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.024703 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca" pod "eb4c6579-8d82-4b02-9680-0a5e57686732" (UID: "eb4c6579-8d82-4b02-9680-0a5e57686732"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:34.026550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.026521 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd" pod "eb4c6579-8d82-4b02-9680-0a5e57686732" (UID: "eb4c6579-8d82-4b02-9680-0a5e57686732"). InnerVolumeSpecName "kube-api-access-2x2dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:34.026992 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.026969 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls" pod "eb4c6579-8d82-4b02-9680-0a5e57686732" (UID: "eb4c6579-8d82-4b02-9680-0a5e57686732"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:34.027093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.026995 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs" pod "eb4c6579-8d82-4b02-9680-0a5e57686732" (UID: "eb4c6579-8d82-4b02-9680-0a5e57686732"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:34.155884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.155323 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:34.155884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.155362 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4c6579-8d82-4b02-9680-0a5e57686732-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:34.155884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.155378 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4c6579-8d82-4b02-9680-0a5e57686732-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:34.155884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.155392 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2dd\" (UniqueName: \"kubernetes.io/projected/eb4c6579-8d82-4b02-9680-0a5e57686732-kube-api-access-2x2dd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:34.330690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.330598 5548 generic.go:356] "Generic (PLEG): container finished" podID="eb4c6579-8d82-4b02-9680-0a5e57686732" containerID="be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44" exitCode=0 Oct 30 01:31:34.330690 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.330654 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" Oct 30 01:31:34.330958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.330687 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" event={"ID":"eb4c6579-8d82-4b02-9680-0a5e57686732","Type":"ContainerDied","Data":"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44"} Oct 30 01:31:34.330958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.330742 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz" event={"ID":"eb4c6579-8d82-4b02-9680-0a5e57686732","Type":"ContainerDied","Data":"70c76c7f09f786ca381196ad112f5f3711f9c8decec8726828f18f57ab73855c"} Oct 30 01:31:34.330958 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.330764 5548 scope.go:122] "RemoveContainer" containerID="be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44" Oct 30 01:31:34.341150 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.341125 5548 scope.go:122] "RemoveContainer" containerID="be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44" Oct 30 01:31:34.341437 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:31:34.341421 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44\": container with ID starting with be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44 not found: ID does not exist" containerID="be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44" Oct 30 01:31:34.341500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.341444 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44"} err="failed to get container status \"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44\": rpc error: code = NotFound desc = could not find container \"be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44\": container with ID starting with be1d8b2daf258cc6e5266bfb5d06296b4856f4a8d5c633fd717197f86f6f5c44 not found: ID does not exist" Oct 30 01:31:34.378476 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.378450 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:31:34.385496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:34.385474 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-886adb-5499cfc45b-tvllz"] Oct 30 01:31:35.263884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:35.263832 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4c6579-8d82-4b02-9680-0a5e57686732" path="/var/lib/kubelet/pods/eb4c6579-8d82-4b02-9680-0a5e57686732/volumes" Oct 30 01:31:35.996521 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:35.996484 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:31:35.996775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:35.996747 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" podUID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" containerName="maas-api" containerID="cri-o://f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915" gracePeriod=30 Oct 30 01:31:36.256254 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.256184 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:31:36.344989 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.344951 5548 generic.go:356] "Generic (PLEG): container finished" podID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" containerID="f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915" exitCode=0 Oct 30 01:31:36.345493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.345000 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" Oct 30 01:31:36.345493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.345036 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" event={"ID":"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a","Type":"ContainerDied","Data":"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915"} Oct 30 01:31:36.345493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.345099 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq" event={"ID":"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a","Type":"ContainerDied","Data":"2029450e4bdfc4defcb675d251b41326439fe13904e01483d821d4a01749371a"} Oct 30 01:31:36.345493 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.345137 5548 scope.go:122] "RemoveContainer" containerID="f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915" Oct 30 01:31:36.356121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.356098 5548 scope.go:122] "RemoveContainer" containerID="f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915" Oct 30 01:31:36.356491 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:31:36.356469 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915\": container with ID starting with f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915 not found: ID does not exist" containerID="f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915" Oct 30 01:31:36.356588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.356501 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915"} err="failed to get container status \"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915\": rpc error: code = NotFound desc = could not find container \"f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915\": container with ID starting with f5eda064ad3e18f4fc2980c121b5262e6d2451384a54bbfa191a40ad426e2915 not found: ID does not exist" Oct 30 01:31:36.382504 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.382477 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\") pod \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " Oct 30 01:31:36.382685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.382526 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\" (UniqueName: \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\") pod \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " Oct 30 01:31:36.382685 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.382564 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\") pod \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " Oct 30 01:31:36.382803 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.382700 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\") pod \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\" (UID: \"e24c0fee-e35a-4b0c-8b8f-538f15cbc68a\") " Oct 30 01:31:36.383069 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.383041 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca" pod "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" (UID: "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:36.383350 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.383325 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:36.384760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.384734 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb" pod "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" (UID: "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a"). InnerVolumeSpecName "kube-api-access-k92tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:36.384852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.384828 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs" pod "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" (UID: "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:36.384917 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.384878 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls" pod "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" (UID: "e24c0fee-e35a-4b0c-8b8f-538f15cbc68a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:36.485370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.485062 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:36.485370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.485093 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:36.485370 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.485103 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92tb\" (UniqueName: \"kubernetes.io/projected/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a-kube-api-access-k92tb\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:36.689707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.689678 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:31:36.698630 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:36.698595 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-d81f0a-b67db7d67-8ftbq"] Oct 30 01:31:37.264079 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:37.264046 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" path="/var/lib/kubelet/pods/e24c0fee-e35a-4b0c-8b8f-538f15cbc68a/volumes" Oct 30 01:31:42.041924 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:42.041829 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh"] Oct 30 01:31:42.051584 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:42.051561 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e7a0ac-8t8nh"] Oct 30 01:31:43.265566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:43.264988 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4553c4-9f56-4590-83ec-a06ff2d0050e" path="/var/lib/kubelet/pods/aa4553c4-9f56-4590-83ec-a06ff2d0050e/volumes" Oct 30 01:31:44.158959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.158926 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:31:44.159172 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.159132 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" containerID="cri-o://bdcf5c171c26c70e1a0e230e0e96ecb2e3baf299983c6ad883eb8ea16177d655" gracePeriod=30 Oct 30 01:31:44.167765 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.167739 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:31:44.167965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.167945 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" containerID="cri-o://7f3d32dea1bf61fc939e8329ee6005cfc8e8ebed12b55c99c8d1cd58e6198e87" gracePeriod=30 Oct 30 01:31:44.179417 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.179391 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:31:44.179583 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:44.179553 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" containerID="cri-o://c32e620c7e909661c21c4ab721c5ff237cb42d6213a878dad470ba8065a68488" gracePeriod=30 Oct 30 01:31:49.428558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.428528 5548 generic.go:356] "Generic (PLEG): container finished" podID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerID="c32e620c7e909661c21c4ab721c5ff237cb42d6213a878dad470ba8065a68488" exitCode=0 Oct 30 01:31:49.429006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.428615 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" event={"ID":"711541f2-3cef-4eda-8ea8-d0f3fa0daeda","Type":"ContainerDied","Data":"c32e620c7e909661c21c4ab721c5ff237cb42d6213a878dad470ba8065a68488"} Oct 30 01:31:49.431001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.430672 5548 generic.go:356] "Generic (PLEG): container finished" podID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerID="bdcf5c171c26c70e1a0e230e0e96ecb2e3baf299983c6ad883eb8ea16177d655" exitCode=0 Oct 30 01:31:49.431001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.430778 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" event={"ID":"81bf83eb-3cc0-49ac-a48b-453f2cf52673","Type":"ContainerDied","Data":"bdcf5c171c26c70e1a0e230e0e96ecb2e3baf299983c6ad883eb8ea16177d655"} Oct 30 01:31:49.431001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.430832 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" event={"ID":"81bf83eb-3cc0-49ac-a48b-453f2cf52673","Type":"ContainerDied","Data":"945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6"} Oct 30 01:31:49.431001 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.430853 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945cf29927c414f6cd3f95d7519cfe36f4514974d146adf6d3078bc8b41fb0c6" Oct 30 01:31:49.433234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.433211 5548 generic.go:356] "Generic (PLEG): container finished" podID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerID="7f3d32dea1bf61fc939e8329ee6005cfc8e8ebed12b55c99c8d1cd58e6198e87" exitCode=0 Oct 30 01:31:49.433340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.433286 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" event={"ID":"991987ed-57be-4005-bb55-691e1c9fa3c9","Type":"ContainerDied","Data":"7f3d32dea1bf61fc939e8329ee6005cfc8e8ebed12b55c99c8d1cd58e6198e87"} Oct 30 01:31:49.434615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.434597 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:31:49.464903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.464853 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:31:49.493608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.493581 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:31:49.538113 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538080 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538122 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538165 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538225 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538270 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538310 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538381 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538446 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.538536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.538489 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\") pod \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\" (UID: \"81bf83eb-3cc0-49ac-a48b-453f2cf52673\") " Oct 30 01:31:49.539369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.539344 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.539565 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.539372 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.539664 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.539565 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.539811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.539781 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.539919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.539507 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.542842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.542814 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "kube-api-access-brvpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.542994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.542843 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.542994 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.542956 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.543953 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.543919 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo" pod "81bf83eb-3cc0-49ac-a48b-453f2cf52673" (UID: "81bf83eb-3cc0-49ac-a48b-453f2cf52673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.570823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.570788 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 01:31:49.640175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640142 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640224 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.640386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640275 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.640386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640330 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.640386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640383 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640417 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640459 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640491 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640529 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640565 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640593 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.640629 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640623 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640663 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640716 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640742 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640770 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640826 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\") pod \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\" (UID: \"711541f2-3cef-4eda-8ea8-d0f3fa0daeda\") " Oct 30 01:31:49.641097 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.640880 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\") pod \"991987ed-57be-4005-bb55-691e1c9fa3c9\" (UID: \"991987ed-57be-4005-bb55-691e1c9fa3c9\") " Oct 30 01:31:49.641423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641159 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.641423 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641411 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641431 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641444 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641457 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641472 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641487 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641533 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641711 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641724 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvpx\" (UniqueName: \"kubernetes.io/projected/81bf83eb-3cc0-49ac-a48b-453f2cf52673-kube-api-access-brvpx\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641738 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bf83eb-3cc0-49ac-a48b-453f2cf52673-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.641798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.641625 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.642094 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.642044 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.642364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.642340 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.643111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.642830 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.643761 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.643456 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.643761 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.643551 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "kube-api-access-69w2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.644202 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.644016 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.644578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.644479 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.644578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.644546 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.644805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.644774 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.645121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.645098 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.645610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.645586 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.646251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.646193 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.646251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.646195 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.646436 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.646390 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "kube-api-access-4g7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.647601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.647569 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo" pod "711541f2-3cef-4eda-8ea8-d0f3fa0daeda" (UID: "711541f2-3cef-4eda-8ea8-d0f3fa0daeda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.647601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.647578 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token" pod "991987ed-57be-4005-bb55-691e1c9fa3c9" (UID: "991987ed-57be-4005-bb55-691e1c9fa3c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743053 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69w2j\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-kube-api-access-69w2j\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743083 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743093 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743101 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991987ed-57be-4005-bb55-691e1c9fa3c9-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743111 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743118 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743127 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743134 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743143 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743151 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743158 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743165 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743174 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743184 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743191 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711541f2-3cef-4eda-8ea8-d0f3fa0daeda-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743199 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7lt\" (UniqueName: \"kubernetes.io/projected/991987ed-57be-4005-bb55-691e1c9fa3c9-kube-api-access-4g7lt\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.743536 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:49.743206 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991987ed-57be-4005-bb55-691e1c9fa3c9-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:31:50.440201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.440116 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" Oct 30 01:31:50.440676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.440120 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s" event={"ID":"711541f2-3cef-4eda-8ea8-d0f3fa0daeda","Type":"ContainerDied","Data":"48f1367b57e90a9fca5f27ac7833e5256e16da70abd86179333f1da15def1670"} Oct 30 01:31:50.440676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.440286 5548 scope.go:122] "RemoveContainer" containerID="c32e620c7e909661c21c4ab721c5ff237cb42d6213a878dad470ba8065a68488" Oct 30 01:31:50.441822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.441798 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" event={"ID":"991987ed-57be-4005-bb55-691e1c9fa3c9","Type":"ContainerDied","Data":"1b69a5d66f10a6b00dc95e106921f3835d835bd0bbfa248086000c8b864b1dec"} Oct 30 01:31:50.441950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.441818 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp" Oct 30 01:31:50.441950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.441829 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s" Oct 30 01:31:50.457554 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.457541 5548 scope.go:122] "RemoveContainer" containerID="7f3d32dea1bf61fc939e8329ee6005cfc8e8ebed12b55c99c8d1cd58e6198e87" Oct 30 01:31:50.515577 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.515544 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:31:50.518529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.518503 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-4q28s"] Oct 30 01:31:50.556993 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.556964 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:31:50.568546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.568519 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-tg72s"] Oct 30 01:31:50.598818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.598793 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:31:50.604846 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:50.604823 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-t6pjp"] Oct 30 01:31:51.264181 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:51.264145 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" path="/var/lib/kubelet/pods/711541f2-3cef-4eda-8ea8-d0f3fa0daeda/volumes" Oct 30 01:31:51.264755 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:51.264733 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" path="/var/lib/kubelet/pods/81bf83eb-3cc0-49ac-a48b-453f2cf52673/volumes" Oct 30 01:31:51.265243 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:31:51.265226 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" path="/var/lib/kubelet/pods/991987ed-57be-4005-bb55-691e1c9fa3c9/volumes" Oct 30 01:31:55.428208 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:31:55.428170 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:32:04.591250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.591205 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" containerID="cri-o://5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b" gracePeriod=45 Oct 30 01:32:04.846061 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.845999 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:32:04.896745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.896715 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.896950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.896771 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.896950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.896885 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.896950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.896930 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.897140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.896988 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\" (UniqueName: \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.897140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.897022 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\" (UniqueName: \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\") pod \"6051abfe-1539-41e7-8877-cb21d2e1eef0\" (UID: \"6051abfe-1539-41e7-8877-cb21d2e1eef0\") " Oct 30 01:32:04.897241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.897183 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.897453 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.897432 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.897557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.897524 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.897557 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.897542 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.899544 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.899498 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.899919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.899891 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:04.900092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.900004 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs" pod "6051abfe-1539-41e7-8877-cb21d2e1eef0" (UID: "6051abfe-1539-41e7-8877-cb21d2e1eef0"). InnerVolumeSpecName "kube-api-access-pt5gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:04.998380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.998336 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.998380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.998376 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6051abfe-1539-41e7-8877-cb21d2e1eef0-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.998580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.998391 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5gs\" (UniqueName: \"kubernetes.io/projected/6051abfe-1539-41e7-8877-cb21d2e1eef0-kube-api-access-pt5gs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.998580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.998407 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.998580 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:04.998420 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6051abfe-1539-41e7-8877-cb21d2e1eef0-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:05.541166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.541073 5548 generic.go:356] "Generic (PLEG): container finished" podID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerID="5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b" exitCode=0 Oct 30 01:32:05.541322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.541224 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" event={"ID":"6051abfe-1539-41e7-8877-cb21d2e1eef0","Type":"ContainerDied","Data":"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b"} Oct 30 01:32:05.541322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.541254 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" event={"ID":"6051abfe-1539-41e7-8877-cb21d2e1eef0","Type":"ContainerDied","Data":"9e28bef2dc312ad1e5d91c7cce9b54d53852e3071881a328a0e797ee3cc6e894"} Oct 30 01:32:05.541322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.541271 5548 scope.go:122] "RemoveContainer" containerID="5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b" Oct 30 01:32:05.541439 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.541387 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf" Oct 30 01:32:05.551019 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.550999 5548 scope.go:122] "RemoveContainer" containerID="5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b" Oct 30 01:32:05.551287 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:05.551264 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b\": container with ID starting with 5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b not found: ID does not exist" containerID="5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b" Oct 30 01:32:05.551365 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.551301 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b"} err="failed to get container status \"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b\": rpc error: code = NotFound desc = could not find container \"5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b\": container with ID starting with 5fba63261c8eef0cba2cac3a26cf5893665c255382c6f85c681b389caf19717b not found: ID does not exist" Oct 30 01:32:05.587222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.587190 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 01:32:05.597002 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:05.596978 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/simulator-w3-82b647-kserve-5f44864fcf-qfpvf"] Oct 30 01:32:07.227071 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:07.227034 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:32:07.227560 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:07.227292 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" containerID="cri-o://1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900" gracePeriod=30 Oct 30 01:32:07.235759 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:07.235734 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:32:07.235974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:07.235953 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" containerID="cri-o://751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0" gracePeriod=30 Oct 30 01:32:07.264431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:07.264402 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" path="/var/lib/kubelet/pods/6051abfe-1539-41e7-8877-cb21d2e1eef0/volumes" Oct 30 01:32:12.516726 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.516697 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:32:12.537353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.537331 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:32:12.579268 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579228 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579295 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579330 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579387 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579447 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579489 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579535 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579573 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579614 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\") pod \"d2847509-9131-43e1-9957-eb18b46773bb\" (UID: \"d2847509-9131-43e1-9957-eb18b46773bb\") " Oct 30 01:32:12.579974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.579790 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.580149 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.580126 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.581745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.581129 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.581745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.581483 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.581745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.581651 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:12.581745 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.581714 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.586032 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.586000 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.590237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.590207 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.590386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.590328 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:12.592071 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.592042 5548 generic.go:356] "Generic (PLEG): container finished" podID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerID="1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900" exitCode=0 Oct 30 01:32:12.592345 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.592209 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" Oct 30 01:32:12.592489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.592371 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" event={"ID":"abc28590-5ca5-43fa-b2c9-29b67c89155d","Type":"ContainerDied","Data":"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900"} Oct 30 01:32:12.592489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.592417 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c" event={"ID":"abc28590-5ca5-43fa-b2c9-29b67c89155d","Type":"ContainerDied","Data":"3979467eeb262289aa01e512d1365a44cbc269222631ed83cf6d154ed585de8a"} Oct 30 01:32:12.592489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.592448 5548 scope.go:122] "RemoveContainer" containerID="1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900" Oct 30 01:32:12.606655 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.606625 5548 generic.go:356] "Generic (PLEG): container finished" podID="d2847509-9131-43e1-9957-eb18b46773bb" containerID="751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0" exitCode=0 Oct 30 01:32:12.606794 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.606706 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" Oct 30 01:32:12.606897 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.606732 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" event={"ID":"d2847509-9131-43e1-9957-eb18b46773bb","Type":"ContainerDied","Data":"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0"} Oct 30 01:32:12.606976 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.606891 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx" event={"ID":"d2847509-9131-43e1-9957-eb18b46773bb","Type":"ContainerDied","Data":"5b0d3e7d37a7a9e583485b8e1773fa379736c6a49b21bea5fbdb0054cfce9430"} Oct 30 01:32:12.608380 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.608352 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr" pod "d2847509-9131-43e1-9957-eb18b46773bb" (UID: "d2847509-9131-43e1-9957-eb18b46773bb"). InnerVolumeSpecName "kube-api-access-zlrmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.617785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.617763 5548 scope.go:122] "RemoveContainer" containerID="1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900" Oct 30 01:32:12.618208 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:12.618182 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900\": container with ID starting with 1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900 not found: ID does not exist" containerID="1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900" Oct 30 01:32:12.618305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.618222 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900"} err="failed to get container status \"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900\": rpc error: code = NotFound desc = could not find container \"1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900\": container with ID starting with 1dceb526cf06e772acc9aafb8ca1eedf0b822152777a928b7861edd3cfe85900 not found: ID does not exist" Oct 30 01:32:12.618305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.618249 5548 scope.go:122] "RemoveContainer" containerID="751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0" Oct 30 01:32:12.633834 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.633813 5548 scope.go:122] "RemoveContainer" containerID="751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0" Oct 30 01:32:12.634303 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:12.634237 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0\": container with ID starting with 751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0 not found: ID does not exist" containerID="751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0" Oct 30 01:32:12.634303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.634273 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0"} err="failed to get container status \"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0\": rpc error: code = NotFound desc = could not find container \"751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0\": container with ID starting with 751735ede73f42207b9daa8c7a2afdc7baf52e1e6c59c5491f08fc896517bba0 not found: ID does not exist" Oct 30 01:32:12.680822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.680766 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.680827 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.680930 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681029 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681061 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681106 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.681401 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681172 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.682135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681826 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.682135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.681917 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\") pod \"abc28590-5ca5-43fa-b2c9-29b67c89155d\" (UID: \"abc28590-5ca5-43fa-b2c9-29b67c89155d\") " Oct 30 01:32:12.682135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.682007 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.683437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.682547 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.683437 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.682950 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:12.683885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.683782 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.684249 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.684207 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686216 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc28590-5ca5-43fa-b2c9-29b67c89155d-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686244 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2847509-9131-43e1-9957-eb18b46773bb-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686260 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2847509-9131-43e1-9957-eb18b46773bb-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686274 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686288 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686302 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686315 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686328 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2847509-9131-43e1-9957-eb18b46773bb-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686343 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrmr\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-kube-api-access-zlrmr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686643 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2847509-9131-43e1-9957-eb18b46773bb-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686663 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686679 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.686741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.686691 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.691280 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.691255 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:12.691394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.691359 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.691472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.691438 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.693262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.693240 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv" pod "abc28590-5ca5-43fa-b2c9-29b67c89155d" (UID: "abc28590-5ca5-43fa-b2c9-29b67c89155d"). InnerVolumeSpecName "kube-api-access-8pvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.788195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.788097 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pvkv\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-kube-api-access-8pvkv\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.788195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.788129 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.788195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.788139 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.788195 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.788147 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc28590-5ca5-43fa-b2c9-29b67c89155d-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.936595 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.936568 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:32:12.940328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.940301 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-vbw6c"] Oct 30 01:32:12.970660 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.970628 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:32:12.974353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:12.974328 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-wp9zx"] Oct 30 01:32:13.263527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:13.263491 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" path="/var/lib/kubelet/pods/abc28590-5ca5-43fa-b2c9-29b67c89155d/volumes" Oct 30 01:32:13.263999 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:13.263983 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2847509-9131-43e1-9957-eb18b46773bb" path="/var/lib/kubelet/pods/d2847509-9131-43e1-9957-eb18b46773bb/volumes" Oct 30 01:32:16.726598 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:16.726567 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 01:32:16.727352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:16.726837 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" podUID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" containerName="maas-api" containerID="cri-o://5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7" gracePeriod=30 Oct 30 01:32:17.006728 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.006696 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 01:32:17.129743 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.129708 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\") pod \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " Oct 30 01:32:17.129963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.129824 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\" (UniqueName: \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\") pod \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " Oct 30 01:32:17.130060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.129963 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\") pod \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " Oct 30 01:32:17.130060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.129999 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\") pod \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\" (UID: \"f06ebece-f45f-4af3-a865-5ff7f6a963ae\") " Oct 30 01:32:17.130427 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.130401 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca" pod "f06ebece-f45f-4af3-a865-5ff7f6a963ae" (UID: "f06ebece-f45f-4af3-a865-5ff7f6a963ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:17.132026 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.131993 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls" pod "f06ebece-f45f-4af3-a865-5ff7f6a963ae" (UID: "f06ebece-f45f-4af3-a865-5ff7f6a963ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:17.132143 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.132101 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq" pod "f06ebece-f45f-4af3-a865-5ff7f6a963ae" (UID: "f06ebece-f45f-4af3-a865-5ff7f6a963ae"). InnerVolumeSpecName "kube-api-access-b92pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:17.132203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.132149 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs" pod "f06ebece-f45f-4af3-a865-5ff7f6a963ae" (UID: "f06ebece-f45f-4af3-a865-5ff7f6a963ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:17.231234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.231203 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:17.231234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.231232 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92pq\" (UniqueName: \"kubernetes.io/projected/f06ebece-f45f-4af3-a865-5ff7f6a963ae-kube-api-access-b92pq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:17.231234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.231242 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06ebece-f45f-4af3-a865-5ff7f6a963ae-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:17.231234 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.231251 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ebece-f45f-4af3-a865-5ff7f6a963ae-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:17.638323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.638226 5548 generic.go:356] "Generic (PLEG): container finished" podID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" containerID="5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7" exitCode=0 Oct 30 01:32:17.638323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.638275 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" Oct 30 01:32:17.638323 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.638302 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" event={"ID":"f06ebece-f45f-4af3-a865-5ff7f6a963ae","Type":"ContainerDied","Data":"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7"} Oct 30 01:32:17.638632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.638363 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql" event={"ID":"f06ebece-f45f-4af3-a865-5ff7f6a963ae","Type":"ContainerDied","Data":"81652ff5a0c36b118a11e016869c6518062a4a93f42a6b6742fb9034b795bd75"} Oct 30 01:32:17.638632 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.638386 5548 scope.go:122] "RemoveContainer" containerID="5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7" Oct 30 01:32:17.649911 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.649891 5548 scope.go:122] "RemoveContainer" containerID="5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7" Oct 30 01:32:17.650265 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:17.650246 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7\": container with ID starting with 5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7 not found: ID does not exist" containerID="5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7" Oct 30 01:32:17.650346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.650279 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7"} err="failed to get container status \"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7\": rpc error: code = NotFound desc = could not find container \"5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7\": container with ID starting with 5da6168daa71baf96499e7a127898de404aee40ba265577265f7ad153149d4c7 not found: ID does not exist" Oct 30 01:32:17.685547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.685518 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 01:32:17.695163 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:17.695136 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-82b647-9589d787b-qj4ql"] Oct 30 01:32:19.263511 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:19.263477 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" path="/var/lib/kubelet/pods/f06ebece-f45f-4af3-a865-5ff7f6a963ae/volumes" Oct 30 01:32:24.131120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:24.131084 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 01:32:24.131683 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:24.131344 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" containerID="cri-o://be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543" gracePeriod=30 Oct 30 01:32:25.427033 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:25.426996 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:32:28.370603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:28.370558 5548 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:32:28.371013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:28.370644 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:32:29.399386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.399328 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 01:32:29.473974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.473944 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.473985 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474005 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474055 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474075 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474118 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474224 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474222 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:29.474430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474309 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:29.474430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474368 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474459 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\") pod \"81e0db78-9933-424d-8460-cd8655b21303\" (UID: \"81e0db78-9933-424d-8460-cd8655b21303\") " Oct 30 01:32:29.474649 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474521 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:29.474786 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474712 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:29.474843 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474800 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:29.474927 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474905 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.474981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474935 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.474981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474950 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.474981 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.474963 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.476543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.476516 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:29.476822 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.476805 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "kube-api-access-8wb5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:29.477012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.476994 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:29.477122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.477107 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy" pod "81e0db78-9933-424d-8460-cd8655b21303" (UID: "81e0db78-9933-424d-8460-cd8655b21303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:29.576357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.576326 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0db78-9933-424d-8460-cd8655b21303-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.576357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.576349 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.576357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.576358 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0db78-9933-424d-8460-cd8655b21303-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.576357 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.576366 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0db78-9933-424d-8460-cd8655b21303-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.576608 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.576374 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wb5d\" (UniqueName: \"kubernetes.io/projected/81e0db78-9933-424d-8460-cd8655b21303-kube-api-access-8wb5d\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.706966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.706850 5548 generic.go:356] "Generic (PLEG): container finished" podID="81e0db78-9933-424d-8460-cd8655b21303" containerID="be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543" exitCode=0 Oct 30 01:32:29.706966 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.706918 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" Oct 30 01:32:29.707219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.706887 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" event={"ID":"81e0db78-9933-424d-8460-cd8655b21303","Type":"ContainerDied","Data":"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543"} Oct 30 01:32:29.707219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.707032 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps" event={"ID":"81e0db78-9933-424d-8460-cd8655b21303","Type":"ContainerDied","Data":"078f39aca87aef53d8d79384b55aa0fd1aa9831934be8e34594b086277758b1f"} Oct 30 01:32:29.707219 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.707063 5548 scope.go:122] "RemoveContainer" containerID="be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543" Oct 30 01:32:29.717517 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.717501 5548 scope.go:122] "RemoveContainer" containerID="be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543" Oct 30 01:32:29.717786 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:29.717763 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543\": container with ID starting with be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543 not found: ID does not exist" containerID="be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543" Oct 30 01:32:29.717835 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.717796 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543"} err="failed to get container status \"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543\": rpc error: code = NotFound desc = could not find container \"be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543\": container with ID starting with be4092cb889a85e175bfca5f733107bb0040bffd4cc5e472de91abe1fb9de543 not found: ID does not exist" Oct 30 01:32:29.757421 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.757390 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 01:32:29.766947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:29.766924 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-w26ps"] Oct 30 01:32:31.055166 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:31.055136 5548 scope.go:122] "RemoveContainer" containerID="bdcf5c171c26c70e1a0e230e0e96ecb2e3baf299983c6ad883eb8ea16177d655" Oct 30 01:32:31.263713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:31.263680 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e0db78-9933-424d-8460-cd8655b21303" path="/var/lib/kubelet/pods/81e0db78-9933-424d-8460-cd8655b21303/volumes" Oct 30 01:32:35.845109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:35.845057 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 01:32:35.845787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:35.845352 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" containerName="maas-api" containerID="cri-o://bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8" gracePeriod=30 Oct 30 01:32:35.950836 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:35.950797 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": dial tcp 10.134.0.42:8443: connect: connection refused" Oct 30 01:32:36.111371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.111312 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 01:32:36.140947 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.140914 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\") pod \"8e877483-0b31-478b-9be6-b7fe52342ded\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " Oct 30 01:32:36.141141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.140955 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\") pod \"8e877483-0b31-478b-9be6-b7fe52342ded\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " Oct 30 01:32:36.141141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.141069 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") pod \"8e877483-0b31-478b-9be6-b7fe52342ded\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " Oct 30 01:32:36.141141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.141109 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\" (UniqueName: \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\") pod \"8e877483-0b31-478b-9be6-b7fe52342ded\" (UID: \"8e877483-0b31-478b-9be6-b7fe52342ded\") " Oct 30 01:32:36.141348 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.141311 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca" pod "8e877483-0b31-478b-9be6-b7fe52342ded" (UID: "8e877483-0b31-478b-9be6-b7fe52342ded"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:36.141567 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.141510 5548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e877483-0b31-478b-9be6-b7fe52342ded-openshift-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.143411 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.143390 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs" pod "8e877483-0b31-478b-9be6-b7fe52342ded" (UID: "8e877483-0b31-478b-9be6-b7fe52342ded"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:36.143991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.143966 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq" pod "8e877483-0b31-478b-9be6-b7fe52342ded" (UID: "8e877483-0b31-478b-9be6-b7fe52342ded"). InnerVolumeSpecName "kube-api-access-x8ssq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:36.144082 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.144013 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls" pod "8e877483-0b31-478b-9be6-b7fe52342ded" (UID: "8e877483-0b31-478b-9be6-b7fe52342ded"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:36.242382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.242350 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-maas-api-tls\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.242382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.242377 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8ssq\" (UniqueName: \"kubernetes.io/projected/8e877483-0b31-478b-9be6-b7fe52342ded-kube-api-access-x8ssq\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.242382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.242388 5548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e877483-0b31-478b-9be6-b7fe52342ded-metrics-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:36.746562 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.746529 5548 generic.go:356] "Generic (PLEG): container finished" podID="8e877483-0b31-478b-9be6-b7fe52342ded" containerID="bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8" exitCode=0 Oct 30 01:32:36.746766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.746574 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" Oct 30 01:32:36.746766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.746612 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" event={"ID":"8e877483-0b31-478b-9be6-b7fe52342ded","Type":"ContainerDied","Data":"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8"} Oct 30 01:32:36.746766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.746666 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b" event={"ID":"8e877483-0b31-478b-9be6-b7fe52342ded","Type":"ContainerDied","Data":"535ff5eb51194c68857e6cbe7fc74b46068ec6ae28424e82d1fdb268c8bf0662"} Oct 30 01:32:36.746766 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.746691 5548 scope.go:122] "RemoveContainer" containerID="bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8" Oct 30 01:32:36.757010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.756992 5548 scope.go:122] "RemoveContainer" containerID="bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8" Oct 30 01:32:36.757291 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:36.757272 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8\": container with ID starting with bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8 not found: ID does not exist" containerID="bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8" Oct 30 01:32:36.757354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.757300 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8"} err="failed to get container status \"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8\": rpc error: code = NotFound desc = could not find container \"bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8\": container with ID starting with bdca1a8ca545f22062bd532e1b6e4b57fd520c1e92cc9cb04e21f0dd85f01ae8 not found: ID does not exist" Oct 30 01:32:36.791314 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.791286 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 01:32:36.801196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:36.801167 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-c4847c-5958f8bbc5-dn66b"] Oct 30 01:32:37.263295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:37.263265 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" path="/var/lib/kubelet/pods/8e877483-0b31-478b-9be6-b7fe52342ded/volumes" Oct 30 01:32:42.951349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:42.951315 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:32:42.951844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:42.951572 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" containerID="cri-o://393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc" gracePeriod=30 Oct 30 01:32:44.852390 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:44.852355 5548 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:32:44.852798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:44.852421 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:32:48.203408 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.203378 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:32:48.356177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356091 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356137 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356177 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356163 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356213 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356267 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356301 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356338 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356371 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356425 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.356489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356441 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\") pod \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\" (UID: \"ab8f95d1-bbf1-48c5-a958-1807c805d7d3\") " Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356658 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356843 5548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356821 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356880 5548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-data\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356851 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:48.356959 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.356904 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.358823 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.358796 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "kube-api-access-psrvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:48.358991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.358923 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:48.359128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.359105 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.359128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.359120 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token" pod "ab8f95d1-bbf1-48c5-a958-1807c805d7d3" (UID: "ab8f95d1-bbf1-48c5-a958-1807c805d7d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:48.458287 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458251 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458287 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458283 5548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-envoy\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458296 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psrvp\" (UniqueName: \"kubernetes.io/projected/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-kube-api-access-psrvp\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458310 5548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istiod-ca-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458322 5548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-workload-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458335 5548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-istio-podinfo\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.458520 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.458345 5548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8f95d1-bbf1-48c5-a958-1807c805d7d3-credential-socket\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.811775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.811737 5548 generic.go:356] "Generic (PLEG): container finished" podID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerID="393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc" exitCode=0 Oct 30 01:32:48.811969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.811787 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" Oct 30 01:32:48.811969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.811813 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" event={"ID":"ab8f95d1-bbf1-48c5-a958-1807c805d7d3","Type":"ContainerDied","Data":"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc"} Oct 30 01:32:48.811969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.811879 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4" event={"ID":"ab8f95d1-bbf1-48c5-a958-1807c805d7d3","Type":"ContainerDied","Data":"4995cf62e2c2453c37e341d8a39aeabb4601cf04d3cb10088ed5a0315a09e271"} Oct 30 01:32:48.811969 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.811907 5548 scope.go:122] "RemoveContainer" containerID="393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc" Oct 30 01:32:48.824161 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.824139 5548 scope.go:122] "RemoveContainer" containerID="393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc" Oct 30 01:32:48.824503 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:48.824483 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc\": container with ID starting with 393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc not found: ID does not exist" containerID="393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc" Oct 30 01:32:48.824592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.824510 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc"} err="failed to get container status \"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc\": rpc error: code = NotFound desc = could not find container \"393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc\": container with ID starting with 393e5cdc335ffae9a5ecb6af439fa9d4835f7884af0c559beec8b02a711b5ddc not found: ID does not exist" Oct 30 01:32:48.869242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.869204 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:32:48.879112 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:48.879080 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-kcht4"] Oct 30 01:32:49.263547 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:32:49.263514 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" path="/var/lib/kubelet/pods/ab8f95d1-bbf1-48c5-a958-1807c805d7d3/volumes" Oct 30 01:32:55.427185 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:32:55.427144 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:33:25.426032 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:33:25.425994 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:33:30.918393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:33:30.918354 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:33:30.927510 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:33:30.927488 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:33:31.114950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:33:31.114917 5548 scope.go:122] "RemoveContainer" containerID="181f251749ea4e5f70bfa820c93959af8879eaffe31d2b8569c25fecee6a2087" Oct 30 01:33:55.434798 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:33:55.434755 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:34:25.427081 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:34:25.427041 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:34:28.850034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.849998 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4"] Oct 30 01:34:28.851236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851207 5548 cpu_manager.go:414] "containerMap: removing container" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" Oct 30 01:34:28.851236 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851234 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851256 5548 cpu_manager.go:414] "containerMap: removing container" podUID="897bdf22-f0c5-48b7-9727-713117e85343" containerName="revoke-keys" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851263 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897bdf22-f0c5-48b7-9727-713117e85343" containerName="revoke-keys" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851272 5548 cpu_manager.go:414] "containerMap: removing container" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851278 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851294 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851301 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851311 5548 cpu_manager.go:414] "containerMap: removing container" podUID="083f1262-dc88-489a-b84d-f0d76c32ab4f" containerName="payload-processing" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851317 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083f1262-dc88-489a-b84d-f0d76c32ab4f" containerName="payload-processing" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851332 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851338 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851347 5548 cpu_manager.go:414] "containerMap: removing container" podUID="392e1d3c-dd4f-48ec-8a69-6ca762010d1a" containerName="revoke-keys" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851353 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392e1d3c-dd4f-48ec-8a69-6ca762010d1a" containerName="revoke-keys" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851367 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851373 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851381 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851388 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851399 5548 cpu_manager.go:414] "containerMap: removing container" podUID="eb4c6579-8d82-4b02-9680-0a5e57686732" containerName="maas-api" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851406 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4c6579-8d82-4b02-9680-0a5e57686732" containerName="maas-api" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851421 5548 cpu_manager.go:414] "containerMap: removing container" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851428 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" Oct 30 01:34:28.851431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851439 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851446 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851455 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851462 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851477 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851484 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851494 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851501 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851513 5548 cpu_manager.go:414] "containerMap: removing container" podUID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851519 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851533 5548 cpu_manager.go:414] "containerMap: removing container" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851540 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851551 5548 cpu_manager.go:414] "containerMap: removing container" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851558 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851575 5548 cpu_manager.go:414] "containerMap: removing container" podUID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851581 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851590 5548 cpu_manager.go:414] "containerMap: removing container" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851597 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851611 5548 cpu_manager.go:414] "containerMap: removing container" podUID="fecfccee-49fb-43d2-8382-d9f4880f78db" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851619 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecfccee-49fb-43d2-8382-d9f4880f78db" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851632 5548 cpu_manager.go:414] "containerMap: removing container" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851639 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851648 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851655 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851665 5548 cpu_manager.go:414] "containerMap: removing container" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851672 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851683 5548 cpu_manager.go:414] "containerMap: removing container" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851689 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851703 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851711 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851726 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851732 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851742 5548 cpu_manager.go:414] "containerMap: removing container" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851749 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851759 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c6504af3-1d50-4b0f-9e02-043033355293" containerName="payload-pre-processing" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851766 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6504af3-1d50-4b0f-9e02-043033355293" containerName="payload-pre-processing" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851779 5548 cpu_manager.go:414] "containerMap: removing container" podUID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851786 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851906 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8f95d1-bbf1-48c5-a958-1807c805d7d3" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851918 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="711541f2-3cef-4eda-8ea8-d0f3fa0daeda" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851928 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a5a2ec-5a4a-48a9-8841-8123894ba61f" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851938 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851946 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851956 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecfccee-49fb-43d2-8382-d9f4880f78db" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851966 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851979 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851987 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6a393e-6d18-4ee2-9c12-bf92f5d7e68e" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.851995 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="991987ed-57be-4005-bb55-691e1c9fa3c9" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852003 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2847509-9131-43e1-9957-eb18b46773bb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852013 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edf8627-750a-48a8-a30d-d8106888e2d1" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852023 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6504af3-1d50-4b0f-9e02-043033355293" containerName="payload-pre-processing" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852032 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35f0cf9-21d8-4272-91fd-6acfc5235e42" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852041 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc28590-5ca5-43fa-b2c9-29b67c89155d" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852051 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="897bdf22-f0c5-48b7-9727-713117e85343" containerName="revoke-keys" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852059 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e0db78-9933-424d-8460-cd8655b21303" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852069 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0707bbd-1bd8-4708-89a1-f2f350fe50f3" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852077 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="083f1262-dc88-489a-b84d-f0d76c32ab4f" containerName="payload-processing" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852086 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852094 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="392e1d3c-dd4f-48ec-8a69-6ca762010d1a" containerName="revoke-keys" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852104 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="d469e1b4-394b-4288-988e-7d5725f3a218" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852114 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6051abfe-1539-41e7-8877-cb21d2e1eef0" containerName="main" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852122 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4c6579-8d82-4b02-9680-0a5e57686732" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852130 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bf83eb-3cc0-49ac-a48b-453f2cf52673" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852139 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8865e1c8-27b0-4549-b818-2068e42c7cbb" containerName="istio-proxy" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852146 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852156 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e877483-0b31-478b-9be6-b7fe52342ded" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852164 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24c0fee-e35a-4b0c-8b8f-538f15cbc68a" containerName="maas-api" Oct 30 01:34:28.852603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.852171 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06ebece-f45f-4af3-a865-5ff7f6a963ae" containerName="maas-api" Oct 30 01:34:28.855702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.855683 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.883118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.883085 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4"] Oct 30 01:34:28.951844 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.951813 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.951943 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a262a9ba-188f-4898-9579-c12ae687dcb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.951982 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952003 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952026 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qhn\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-kube-api-access-h6qhn\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952046 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952168 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952208 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952201 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:28.952286 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:28.952218 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.052988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.052956 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.052988 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.052990 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053013 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qhn\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-kube-api-access-h6qhn\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053146 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053248 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053233 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053293 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053278 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053312 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053376 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053349 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053404 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053429 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a262a9ba-188f-4898-9579-c12ae687dcb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053494 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053484 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053597 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.053783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.053697 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.054395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.054364 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a262a9ba-188f-4898-9579-c12ae687dcb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.055627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.055606 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.055783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.055764 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.070132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.070110 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.070340 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.070319 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qhn\" (UniqueName: \"kubernetes.io/projected/a262a9ba-188f-4898-9579-c12ae687dcb6-kube-api-access-h6qhn\") pod \"maas-default-gateway-openshift-default-67bf986f99-b88g4\" (UID: \"a262a9ba-188f-4898-9579-c12ae687dcb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.170006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.169983 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:29.485145 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.485124 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4"] Oct 30 01:34:29.487481 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:34:29.487456 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda262a9ba_188f_4898_9579_c12ae687dcb6.slice/crio-ebb7e4cad8941d7ab8fb1ba74f228c11a9c4f7b7084bc8bf99fe1bc850614794 WatchSource:0}: Error finding container ebb7e4cad8941d7ab8fb1ba74f228c11a9c4f7b7084bc8bf99fe1bc850614794: Status 404 returned error can't find the container with id ebb7e4cad8941d7ab8fb1ba74f228c11a9c4f7b7084bc8bf99fe1bc850614794 Oct 30 01:34:29.489627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.489593 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:29.489724 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.489658 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:29.489724 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:29.489688 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:30.339735 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:30.339685 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" event={"ID":"a262a9ba-188f-4898-9579-c12ae687dcb6","Type":"ContainerStarted","Data":"80885b5fcff1a3cf08f9ab5859ddade0093518fc597ac6b79891092fa3e4e6ff"} Oct 30 01:34:30.340185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:30.339751 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" event={"ID":"a262a9ba-188f-4898-9579-c12ae687dcb6","Type":"ContainerStarted","Data":"ebb7e4cad8941d7ab8fb1ba74f228c11a9c4f7b7084bc8bf99fe1bc850614794"} Oct 30 01:34:30.399397 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:30.399341 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podStartSLOduration=2.399321879 podStartE2EDuration="2.399321879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:30.387321242 +0000 UTC m=+3061.229881835" watchObservedRunningTime="2026-10-30 01:34:30.399321879 +0000 UTC m=+3061.241882471" Oct 30 01:34:31.171884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:31.171010 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:31.192422 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:31.192384 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:31.192739 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:31.192700 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:32.181885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:32.173230 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:32.181885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:32.173291 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:33.173251 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:33.173215 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:33.173545 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:33.173517 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:34.173264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:34.173223 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:34.173837 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:34.173804 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:35.173885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:35.171208 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:35.173885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:35.171271 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:36.171102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:36.171053 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:36.171326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:36.171125 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:37.170813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:37.170776 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 01:34:37.171263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:37.170842 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 01:34:39.212064 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:39.212015 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:39.212523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:39.212093 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:40.212777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:40.212685 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:40.212777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:40.212752 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:41.254023 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:41.253990 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:41.254529 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:41.254058 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:42.172919 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:42.172889 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:42.173241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:42.173193 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:42.173847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:42.173827 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" Oct 30 01:34:43.872588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.872557 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt"] Oct 30 01:34:43.873044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.873029 5548 cpu_manager.go:414] "containerMap: removing container" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:43.873101 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.873049 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:43.873142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.873128 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="8988d9f8-b11b-454c-b1aa-f859e56d4389" containerName="cleanup" Oct 30 01:34:43.876518 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.876498 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.902242 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.902218 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt"] Oct 30 01:34:43.922898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.922850 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.922898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.922900 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnftl\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-kube-api-access-jnftl\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.922973 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923014 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923038 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923060 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923082 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923102 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:43.923147 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:43.923146 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024029 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024073 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024138 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024207 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024230 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnftl\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-kube-api-access-jnftl\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024272 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024320 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024347 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024495 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024380 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024686 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024661 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024708 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024842 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024819 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.024977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.024957 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.025153 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.025136 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.026783 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.026760 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.027108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.027091 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.047246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.047227 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnftl\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-kube-api-access-jnftl\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.048617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.048593 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7c855c-7b8a-44a4-b19b-dfea17d5cda5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-f5pvt\" (UID: \"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.189785 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.189762 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:44.501734 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.501708 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt"] Oct 30 01:34:44.503345 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:34:44.503315 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7c855c_7b8a_44a4_b19b_dfea17d5cda5.slice/crio-c063be55288ea3d7e72b4b4db9f198d447ce0ced9f1ac3a2464b3bf97a346ba0 WatchSource:0}: Error finding container c063be55288ea3d7e72b4b4db9f198d447ce0ced9f1ac3a2464b3bf97a346ba0: Status 404 returned error can't find the container with id c063be55288ea3d7e72b4b4db9f198d447ce0ced9f1ac3a2464b3bf97a346ba0 Oct 30 01:34:44.505431 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.505399 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:44.505506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.505469 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:44.505506 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:44.505499 5548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891992Ki","pods":"250"} Oct 30 01:34:45.427592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:45.427553 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" event={"ID":"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5","Type":"ContainerStarted","Data":"77320c50847095b78cb22b14f84641b14a184d2a701daac4f84efdf1b1889b1b"} Oct 30 01:34:45.428108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:45.427614 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" event={"ID":"3a7c855c-7b8a-44a4-b19b-dfea17d5cda5","Type":"ContainerStarted","Data":"c063be55288ea3d7e72b4b4db9f198d447ce0ced9f1ac3a2464b3bf97a346ba0"} Oct 30 01:34:46.192056 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:46.192018 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:46.193646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:46.193615 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:46.193787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:46.193664 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:47.192309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:47.192275 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:47.192887 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:47.192336 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:48.194884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:48.192959 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:48.194884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:48.193023 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:49.191886 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:49.191819 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:49.192225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:49.192180 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:50.195885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:50.192369 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:50.195885 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:50.192448 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:51.190979 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:51.190939 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 01:34:51.191178 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:51.190999 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 01:34:53.191275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:53.191219 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:53.191809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:53.191289 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:54.232041 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.231997 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:54.232593 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.232051 5548 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:54.739013 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.738974 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:54.739434 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.739396 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:54.740204 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.740188 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" Oct 30 01:34:54.788757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:34:54.788705 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podStartSLOduration=11.788689311 podStartE2EDuration="11.788689311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:45.498826166 +0000 UTC m=+3076.341386758" watchObservedRunningTime="2026-10-30 01:34:54.788689311 +0000 UTC m=+3085.631249990" Oct 30 01:34:55.427080 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:34:55.427041 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:35:10.026687 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:35:10.026648 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz"] Oct 30 01:35:10.036702 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:35:10.036672 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600aaa78-cqbfz"] Oct 30 01:35:11.265556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:35:11.265521 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897bdf22-f0c5-48b7-9727-713117e85343" path="/var/lib/kubelet/pods/897bdf22-f0c5-48b7-9727-713117e85343/volumes" Oct 30 01:35:25.431841 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:35:25.431774 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:35:47.376015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:35:47.375970 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-7r8hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:35:47.376798 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:35:47.376766 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-7r8hn" podUID="9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:35:55.426399 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:35:55.426358 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:36:03.097399 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.097361 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:36:03.097968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.097626 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" podUID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" containerName="discovery" containerID="cri-o://0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775" gracePeriod=30 Oct 30 01:36:03.350144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.350085 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:36:03.498253 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498160 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498289 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498332 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498356 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498468 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498497 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.498573 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.498546 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\") pod \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\" (UID: \"b5efbe9d-b694-4bf8-9183-526e58a64f7d\") " Oct 30 01:36:03.499245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.499210 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:03.501568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.501497 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:03.501833 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.501805 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:03.502923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.502841 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:03.502923 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.502853 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:03.503057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.503020 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:03.503098 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.503074 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl" pod "b5efbe9d-b694-4bf8-9183-526e58a64f7d" (UID: "b5efbe9d-b694-4bf8-9183-526e58a64f7d"). InnerVolumeSpecName "kube-api-access-tgznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:03.600120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600092 5548 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5efbe9d-b694-4bf8-9183-526e58a64f7d-local-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600120 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600118 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgznl\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-kube-api-access-tgznl\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600129 5548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-ca-configmap\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600138 5548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-token\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600147 5548 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-kubeconfig\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600154 5548 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-cacerts\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.600341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.600161 5548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5efbe9d-b694-4bf8-9183-526e58a64f7d-istio-csr-dns-cert\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:36:03.920662 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.920632 5548 generic.go:356] "Generic (PLEG): container finished" podID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" containerID="0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775" exitCode=0 Oct 30 01:36:03.920880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.920680 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" Oct 30 01:36:03.920880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.920694 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" event={"ID":"b5efbe9d-b694-4bf8-9183-526e58a64f7d","Type":"ContainerDied","Data":"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775"} Oct 30 01:36:03.920880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.920733 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm" event={"ID":"b5efbe9d-b694-4bf8-9183-526e58a64f7d","Type":"ContainerDied","Data":"0cc0e29a800b54fdc3fccc9c8fd538447a915a55da34e9cb5496180d77e74988"} Oct 30 01:36:03.920880 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.920752 5548 scope.go:122] "RemoveContainer" containerID="0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775" Oct 30 01:36:03.933689 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.933669 5548 scope.go:122] "RemoveContainer" containerID="0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775" Oct 30 01:36:03.934040 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:36:03.934016 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775\": container with ID starting with 0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775 not found: ID does not exist" containerID="0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775" Oct 30 01:36:03.934118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.934051 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775"} err="failed to get container status \"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775\": rpc error: code = NotFound desc = could not find container \"0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775\": container with ID starting with 0024ebef00fa9100d02c2b1a6e820bd41fb23b6b3e5b1939d7bc176b0b226775 not found: ID does not exist" Oct 30 01:36:03.975682 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.975656 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:36:03.985450 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:03.985424 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xq4mm"] Oct 30 01:36:05.263443 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:05.263408 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" path="/var/lib/kubelet/pods/b5efbe9d-b694-4bf8-9183-526e58a64f7d/volumes" Oct 30 01:36:25.426362 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:36:25.426321 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:36:28.244889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:28.231510 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:36:28.244889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:28.231606 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:36:31.161346 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:31.161302 5548 scope.go:122] "RemoveContainer" containerID="3bb6fd546f8d3b2a6451c60da784aa0ff7867d237c96e2dd6fe468fa2e9d7f07" Oct 30 01:36:31.170238 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:36:31.170219 5548 scope.go:122] "RemoveContainer" containerID="1118fecabe981b72d630cb646c00accfd055fcf29ead3d8cde7cab037c7136c5" Oct 30 01:36:55.428056 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:36:55.428018 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:37:25.426279 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:37:25.426240 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:37:55.427208 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:37:55.427115 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:37:58.071374 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.071336 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:37:58.071974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.071954 5548 cpu_manager.go:414] "containerMap: removing container" podUID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" containerName="discovery" Oct 30 01:37:58.072077 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.071976 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" containerName="discovery" Oct 30 01:37:58.072138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.072112 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5efbe9d-b694-4bf8-9183-526e58a64f7d" containerName="discovery" Oct 30 01:37:58.075192 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.075170 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.089465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.089443 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-37ba51\"/\"kube-root-ca.crt\"" Oct 30 01:37:58.099259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.099234 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-37ba51\"/\"openshift-service-ca.crt\"" Oct 30 01:37:58.102163 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.102137 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:37:58.104182 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.104159 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-37ba51\"/\"default-dockercfg-lj572\"" Oct 30 01:37:58.104425 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.104395 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-37ba51\"/\"simulator-single-main-37ba51-kserve-self-signed-certs\"" Oct 30 01:37:58.180194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180165 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.180194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180199 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.180428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180218 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.180428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180235 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.180428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180391 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.180556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.180445 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvgn\" (UniqueName: \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282046 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282010 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282046 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282040 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282059 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282081 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282105 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282288 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282139 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvgn\" (UniqueName: \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282522 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282623 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.282670 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.282645 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.285259 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.285235 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.285395 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.285379 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.300057 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.300036 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvgn\" (UniqueName: \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\") pod \"simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.388456 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.388294 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:37:58.692492 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:58.692459 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:37:58.694938 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:37:58.694916 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod650837a0_a0e5_4281_9d32_7b1e96bbfa59.slice/crio-6cf4ee5184b620c6edaf2f12cfbd245c18ef843a71978defe702331074450edf WatchSource:0}: Error finding container 6cf4ee5184b620c6edaf2f12cfbd245c18ef843a71978defe702331074450edf: Status 404 returned error can't find the container with id 6cf4ee5184b620c6edaf2f12cfbd245c18ef843a71978defe702331074450edf Oct 30 01:37:59.551184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:59.551129 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" event={"ID":"650837a0-a0e5-4281-9d32-7b1e96bbfa59","Type":"ContainerStarted","Data":"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8"} Oct 30 01:37:59.551602 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:59.551191 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" event={"ID":"650837a0-a0e5-4281-9d32-7b1e96bbfa59","Type":"ContainerStarted","Data":"6cf4ee5184b620c6edaf2f12cfbd245c18ef843a71978defe702331074450edf"} Oct 30 01:37:59.589566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:37:59.589510 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" podStartSLOduration=1.589493399 podStartE2EDuration="1.589493399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:59.5864855 +0000 UTC m=+3270.429046094" watchObservedRunningTime="2026-10-30 01:37:59.589493399 +0000 UTC m=+3270.432053992" Oct 30 01:38:00.389160 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:00.389125 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:08.388615 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:08.388574 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:08.401244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:08.401223 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:08.611956 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:08.611929 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:21.580341 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.580308 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j"] Oct 30 01:38:21.583965 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.583939 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.596847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.596818 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:21.597000 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.596918 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:21.606777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.606728 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:21.606777 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.606757 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:38:21.611163 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.611137 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j"] Oct 30 01:38:21.710269 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.710236 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.710457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.710281 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.710457 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.710404 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwmc\" (UniqueName: \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.811262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.811228 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.811446 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.811274 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.811446 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.811336 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwmc\" (UniqueName: \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.811920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.811899 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.813920 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.813901 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.829566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.829546 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwmc\" (UniqueName: \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\") pod \"maas-api-revoke-sub-f89ac609-sr46j\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:21.897237 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:21.897215 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:22.203537 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:22.203513 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j"] Oct 30 01:38:22.205399 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:38:22.205367 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f1fd05_19d8_4dd4_9ec2_1c2a66480711.slice/crio-c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b WatchSource:0}: Error finding container c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b: Status 404 returned error can't find the container with id c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b Oct 30 01:38:22.677349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:22.677310 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" event={"ID":"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711","Type":"ContainerStarted","Data":"1a5415d6de8f59ab5aa1e67549557e06b1726ca2555384746345408ef8993e35"} Oct 30 01:38:22.677779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:22.677377 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" event={"ID":"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711","Type":"ContainerStarted","Data":"c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b"} Oct 30 01:38:22.713415 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:22.713341 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" podStartSLOduration=1.7133263859999999 podStartE2EDuration="1.713326386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:22.712141259 +0000 UTC m=+3293.554701853" watchObservedRunningTime="2026-10-30 01:38:22.713326386 +0000 UTC m=+3293.555886981" Oct 30 01:38:23.684036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:23.683997 5548 generic.go:356] "Generic (PLEG): container finished" podID="a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" containerID="1a5415d6de8f59ab5aa1e67549557e06b1726ca2555384746345408ef8993e35" exitCode=0 Oct 30 01:38:23.684458 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:23.684074 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" event={"ID":"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711","Type":"ContainerDied","Data":"1a5415d6de8f59ab5aa1e67549557e06b1726ca2555384746345408ef8993e35"} Oct 30 01:38:24.822524 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.822501 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:24.942680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.942599 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\") pod \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " Oct 30 01:38:24.942876 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.942709 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\") pod \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " Oct 30 01:38:24.942876 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.942736 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\" (UniqueName: \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\") pod \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\" (UID: \"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711\") " Oct 30 01:38:24.943171 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.943145 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca" pod "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" (UID: "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:24.944879 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.944844 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca" pod "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" (UID: "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:24.945047 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:24.945027 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc" pod "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" (UID: "a5f1fd05-19d8-4dd4-9ec2-1c2a66480711"). InnerVolumeSpecName "kube-api-access-jmwmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:25.044273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.044237 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:25.044273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.044269 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:25.044273 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.044279 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwmc\" (UniqueName: \"kubernetes.io/projected/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711-kube-api-access-jmwmc\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:25.427789 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:38:25.427753 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:38:25.698155 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.698114 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" event={"ID":"a5f1fd05-19d8-4dd4-9ec2-1c2a66480711","Type":"ContainerDied","Data":"c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b"} Oct 30 01:38:25.698364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.698221 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c01ef896c7170d967d6b7a9984d92bb7372604fe4a02ffdb80a897f87f7f081b" Oct 30 01:38:25.698364 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:25.698306 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j" Oct 30 01:38:30.956502 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:30.956470 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:38:30.965997 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:30.965974 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:38:32.733028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.732996 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8"] Oct 30 01:38:32.733639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.733492 5548 cpu_manager.go:414] "containerMap: removing container" podUID="a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" containerName="revoke-keys" Oct 30 01:38:32.733639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.733508 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" containerName="revoke-keys" Oct 30 01:38:32.733639 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.733616 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" containerName="revoke-keys" Oct 30 01:38:32.736721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.736703 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.760884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.760503 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:32.760884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.760583 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:32.760884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.760593 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:32.760884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.760666 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:38:32.762695 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.762678 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8"] Oct 30 01:38:32.814062 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.814036 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch672\" (UniqueName: \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.814222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.814087 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.814222 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.814108 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.914692 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.914638 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.914852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.914768 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch672\" (UniqueName: \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.914852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.914814 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.915381 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.915362 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.917164 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.917141 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:32.932665 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:32.932640 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch672\" (UniqueName: \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\") pod \"maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:33.049206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:33.049117 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:33.357292 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:33.357257 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8"] Oct 30 01:38:33.359302 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:38:33.359276 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604bd569_34c1_41bc_a78e_fcf57d974f53.slice/crio-9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064 WatchSource:0}: Error finding container 9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064: Status 404 returned error can't find the container with id 9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064 Oct 30 01:38:33.740802 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:33.740763 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" event={"ID":"604bd569-34c1-41bc-a78e-fcf57d974f53","Type":"ContainerStarted","Data":"2f3981f4a4029dad59a8ea8a82caa026b859d402b9ada18d9870a2336069b10f"} Oct 30 01:38:33.741240 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:33.740832 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" event={"ID":"604bd569-34c1-41bc-a78e-fcf57d974f53","Type":"ContainerStarted","Data":"9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064"} Oct 30 01:38:33.782371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:33.782302 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" podStartSLOduration=1.782284084 podStartE2EDuration="1.782284084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:33.779845337 +0000 UTC m=+3304.622405931" watchObservedRunningTime="2026-10-30 01:38:33.782284084 +0000 UTC m=+3304.624844677" Oct 30 01:38:34.747291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:34.747256 5548 generic.go:356] "Generic (PLEG): container finished" podID="604bd569-34c1-41bc-a78e-fcf57d974f53" containerID="2f3981f4a4029dad59a8ea8a82caa026b859d402b9ada18d9870a2336069b10f" exitCode=0 Oct 30 01:38:34.747673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:34.747341 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" event={"ID":"604bd569-34c1-41bc-a78e-fcf57d974f53","Type":"ContainerDied","Data":"2f3981f4a4029dad59a8ea8a82caa026b859d402b9ada18d9870a2336069b10f"} Oct 30 01:38:35.884100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.884075 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:35.945519 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.945494 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\") pod \"604bd569-34c1-41bc-a78e-fcf57d974f53\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " Oct 30 01:38:35.945698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.945638 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\" (UniqueName: \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\") pod \"604bd569-34c1-41bc-a78e-fcf57d974f53\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " Oct 30 01:38:35.945698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.945688 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\") pod \"604bd569-34c1-41bc-a78e-fcf57d974f53\" (UID: \"604bd569-34c1-41bc-a78e-fcf57d974f53\") " Oct 30 01:38:35.945921 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.945900 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca" pod "604bd569-34c1-41bc-a78e-fcf57d974f53" (UID: "604bd569-34c1-41bc-a78e-fcf57d974f53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:35.946048 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.946022 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:35.947893 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.947853 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672" pod "604bd569-34c1-41bc-a78e-fcf57d974f53" (UID: "604bd569-34c1-41bc-a78e-fcf57d974f53"). InnerVolumeSpecName "kube-api-access-ch672". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:35.948010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:35.947892 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca" pod "604bd569-34c1-41bc-a78e-fcf57d974f53" (UID: "604bd569-34c1-41bc-a78e-fcf57d974f53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:36.047018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:36.046951 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch672\" (UniqueName: \"kubernetes.io/projected/604bd569-34c1-41bc-a78e-fcf57d974f53-kube-api-access-ch672\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:36.047018 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:36.046975 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604bd569-34c1-41bc-a78e-fcf57d974f53-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:36.758140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:36.758116 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" Oct 30 01:38:36.758328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:36.758152 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8" event={"ID":"604bd569-34c1-41bc-a78e-fcf57d974f53","Type":"ContainerDied","Data":"9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064"} Oct 30 01:38:36.758328 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:36.758203 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea5d73e0bab639c9f2d9fc4572f657e45380dc30b38d0c205b3179ac13a4064" Oct 30 01:38:37.522418 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:37.522387 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:38:46.376705 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:46.376668 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8"] Oct 30 01:38:46.386303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:46.386272 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-37ba51-t99v8"] Oct 30 01:38:47.262839 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:47.262804 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604bd569-34c1-41bc-a78e-fcf57d974f53" path="/var/lib/kubelet/pods/604bd569-34c1-41bc-a78e-fcf57d974f53/volumes" Oct 30 01:38:52.543074 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.543002 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerName="main" containerID="cri-o://43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8" gracePeriod=45 Oct 30 01:38:52.600361 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.600330 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": dial tcp 10.134.0.169:8000: connect: connection refused" Oct 30 01:38:52.787460 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.787432 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:52.846055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.845948 5548 generic.go:356] "Generic (PLEG): container finished" podID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerID="43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8" exitCode=0 Oct 30 01:38:52.846055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.846007 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" event={"ID":"650837a0-a0e5-4281-9d32-7b1e96bbfa59","Type":"ContainerDied","Data":"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8"} Oct 30 01:38:52.846055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.845990 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" Oct 30 01:38:52.846055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.846041 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb" event={"ID":"650837a0-a0e5-4281-9d32-7b1e96bbfa59","Type":"ContainerDied","Data":"6cf4ee5184b620c6edaf2f12cfbd245c18ef843a71978defe702331074450edf"} Oct 30 01:38:52.846055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.846057 5548 scope.go:122] "RemoveContainer" containerID="43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8" Oct 30 01:38:52.856581 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.855702 5548 scope.go:122] "RemoveContainer" containerID="43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8" Oct 30 01:38:52.856703 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:38:52.856658 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8\": container with ID starting with 43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8 not found: ID does not exist" containerID="43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8" Oct 30 01:38:52.856760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.856697 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8"} err="failed to get container status \"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8\": rpc error: code = NotFound desc = could not find container \"43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8\": container with ID starting with 43e3967b14e90fcc1561d328c909cb2f541951d780cc6bfe94bd3d7769c1a4a8 not found: ID does not exist" Oct 30 01:38:52.902185 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902149 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\" (UniqueName: \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902247 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\" (UniqueName: \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902300 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902424 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902489 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902460 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902504 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\") pod \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\" (UID: \"650837a0-a0e5-4281-9d32-7b1e96bbfa59\") " Oct 30 01:38:52.902610 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902562 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:52.902794 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902762 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:52.902898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902811 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:52.902973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902919 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:52.902973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902939 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:52.902973 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.902953 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:52.904775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.904748 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:52.904983 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.904955 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:52.905068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:52.904985 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn" pod "650837a0-a0e5-4281-9d32-7b1e96bbfa59" (UID: "650837a0-a0e5-4281-9d32-7b1e96bbfa59"). InnerVolumeSpecName "kube-api-access-qcvgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:53.004302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.004275 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650837a0-a0e5-4281-9d32-7b1e96bbfa59-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:53.004302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.004300 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650837a0-a0e5-4281-9d32-7b1e96bbfa59-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:53.004302 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.004309 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvgn\" (UniqueName: \"kubernetes.io/projected/650837a0-a0e5-4281-9d32-7b1e96bbfa59-kube-api-access-qcvgn\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:38:53.190619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.190587 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:38:53.195980 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.195953 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-37ba51/simulator-single-main-37ba51-kserve-56f58f6c8c-56hjb"] Oct 30 01:38:53.263217 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:38:53.263185 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" path="/var/lib/kubelet/pods/650837a0-a0e5-4281-9d32-7b1e96bbfa59/volumes" Oct 30 01:38:55.426614 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:38:55.426579 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:39:25.426733 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:39:25.426695 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:39:55.427091 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:39:55.427052 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:40:25.427110 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:40:25.427068 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:40:55.427004 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:40:55.426924 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:41:25.427577 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:41:25.427531 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:41:27.138805 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.138773 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139202 5548 cpu_manager.go:414] "containerMap: removing container" podUID="604bd569-34c1-41bc-a78e-fcf57d974f53" containerName="revoke-keys" Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139214 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604bd569-34c1-41bc-a78e-fcf57d974f53" containerName="revoke-keys" Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139227 5548 cpu_manager.go:414] "containerMap: removing container" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerName="main" Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139232 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerName="main" Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139293 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="604bd569-34c1-41bc-a78e-fcf57d974f53" containerName="revoke-keys" Oct 30 01:41:27.139300 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.139300 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="650837a0-a0e5-4281-9d32-7b1e96bbfa59" containerName="main" Oct 30 01:41:27.142568 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.142545 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155338 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155309 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155465 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155355 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155532 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155477 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155631 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155608 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155684 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155669 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.155721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.155700 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8kc\" (UniqueName: \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.165322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.165293 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b63f8a\"/\"simulator-single-main-b63f8a-kserve-self-signed-certs\"" Oct 30 01:41:27.165322 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.165315 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b63f8a\"/\"default-dockercfg-cw5z2\"" Oct 30 01:41:27.165600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.165328 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b63f8a\"/\"kube-root-ca.crt\"" Oct 30 01:41:27.165600 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.165354 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b63f8a\"/\"openshift-service-ca.crt\"" Oct 30 01:41:27.167606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.167582 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:41:27.256244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256205 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256244 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256241 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8kc\" (UniqueName: \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256296 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256314 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256342 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256531 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256418 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256712 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.256764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.256750 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.257040 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.257017 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.259258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.259235 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.259382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.259334 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.273563 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.273535 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8kc\" (UniqueName: \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\") pod \"simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.457349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.457295 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:27.774890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:27.774845 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:41:27.777728 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:41:27.777698 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ea8d42d_87c2_4331_a8ef_c6297bc73571.slice/crio-c1e458e0c54cf166bcc1a0e5b8d7f20823a0563efef617f1b8529fe037583a42 WatchSource:0}: Error finding container c1e458e0c54cf166bcc1a0e5b8d7f20823a0563efef617f1b8529fe037583a42: Status 404 returned error can't find the container with id c1e458e0c54cf166bcc1a0e5b8d7f20823a0563efef617f1b8529fe037583a42 Oct 30 01:41:28.724800 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:28.724763 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" event={"ID":"7ea8d42d-87c2-4331-a8ef-c6297bc73571","Type":"ContainerStarted","Data":"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6"} Oct 30 01:41:28.725225 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:28.724810 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" event={"ID":"7ea8d42d-87c2-4331-a8ef-c6297bc73571","Type":"ContainerStarted","Data":"c1e458e0c54cf166bcc1a0e5b8d7f20823a0563efef617f1b8529fe037583a42"} Oct 30 01:41:28.766475 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:28.766420 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" podStartSLOduration=1.7664049720000001 podStartE2EDuration="1.766404972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:28.764098699 +0000 UTC m=+3479.606659293" watchObservedRunningTime="2026-10-30 01:41:28.766404972 +0000 UTC m=+3479.608965564" Oct 30 01:41:29.458326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:29.458294 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:37.457721 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:37.457688 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:37.470533 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:37.470507 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:37.783176 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:37.783094 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:41:45.516811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.516766 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs"] Oct 30 01:41:45.520451 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.520433 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.543369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.543346 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:45.543646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.543622 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:45.543803 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.543655 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:41:45.543803 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.543659 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:45.546141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.546118 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs"] Oct 30 01:41:45.627044 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.627009 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.627246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.627062 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.627246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.627158 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkfd\" (UniqueName: \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.728265 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.728231 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.728471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.728274 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkfd\" (UniqueName: \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.728471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.728347 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.729030 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.729005 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.731021 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.731000 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.746889 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.746845 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkfd\" (UniqueName: \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\") pod \"maas-api-revoke-sub-54a40efb-2n2hs\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:45.834122 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:45.834048 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:46.130555 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:46.130532 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs"] Oct 30 01:41:46.133015 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:41:46.132979 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50d870c_10db_4f47_9809_de6b8c368c84.slice/crio-e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56 WatchSource:0}: Error finding container e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56: Status 404 returned error can't find the container with id e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56 Oct 30 01:41:46.820459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:46.820419 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" event={"ID":"c50d870c-10db-4f47-9809-de6b8c368c84","Type":"ContainerStarted","Data":"7990a6f905214035ee1a223ef42d8990c79143cef53d3b72554d30776503f2aa"} Oct 30 01:41:46.820909 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:46.820499 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" event={"ID":"c50d870c-10db-4f47-9809-de6b8c368c84","Type":"ContainerStarted","Data":"e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56"} Oct 30 01:41:46.858313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:46.858257 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" podStartSLOduration=1.8582410249999999 podStartE2EDuration="1.858241025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:46.856204127 +0000 UTC m=+3497.698764721" watchObservedRunningTime="2026-10-30 01:41:46.858241025 +0000 UTC m=+3497.700801611" Oct 30 01:41:47.827331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:47.827297 5548 generic.go:356] "Generic (PLEG): container finished" podID="c50d870c-10db-4f47-9809-de6b8c368c84" containerID="7990a6f905214035ee1a223ef42d8990c79143cef53d3b72554d30776503f2aa" exitCode=0 Oct 30 01:41:47.827331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:47.827335 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" event={"ID":"c50d870c-10db-4f47-9809-de6b8c368c84","Type":"ContainerDied","Data":"7990a6f905214035ee1a223ef42d8990c79143cef53d3b72554d30776503f2aa"} Oct 30 01:41:48.968428 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:48.968402 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:49.057387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.057360 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\") pod \"c50d870c-10db-4f47-9809-de6b8c368c84\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " Oct 30 01:41:49.057558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.057442 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\") pod \"c50d870c-10db-4f47-9809-de6b8c368c84\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " Oct 30 01:41:49.057558 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.057502 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\" (UniqueName: \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\") pod \"c50d870c-10db-4f47-9809-de6b8c368c84\" (UID: \"c50d870c-10db-4f47-9809-de6b8c368c84\") " Oct 30 01:41:49.057770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.057749 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca" pod "c50d870c-10db-4f47-9809-de6b8c368c84" (UID: "c50d870c-10db-4f47-9809-de6b8c368c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:49.059688 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.059666 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd" pod "c50d870c-10db-4f47-9809-de6b8c368c84" (UID: "c50d870c-10db-4f47-9809-de6b8c368c84"). InnerVolumeSpecName "kube-api-access-5pkfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:49.059756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.059668 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca" pod "c50d870c-10db-4f47-9809-de6b8c368c84" (UID: "c50d870c-10db-4f47-9809-de6b8c368c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:49.158433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.158410 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:41:49.158433 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.158435 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50d870c-10db-4f47-9809-de6b8c368c84-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:41:49.158609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.158444 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkfd\" (UniqueName: \"kubernetes.io/projected/c50d870c-10db-4f47-9809-de6b8c368c84-kube-api-access-5pkfd\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:41:49.838414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.838382 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" event={"ID":"c50d870c-10db-4f47-9809-de6b8c368c84","Type":"ContainerDied","Data":"e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56"} Oct 30 01:41:49.838414 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.838406 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs" Oct 30 01:41:49.838652 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:49.838429 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f41c3475d57b49fc06018cdc337acd49b50249e94686dee8f9337a6cdfba56" Oct 30 01:41:55.426570 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:41:55.426528 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:41:56.539305 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.539264 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff"] Oct 30 01:41:56.539934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.539913 5548 cpu_manager.go:414] "containerMap: removing container" podUID="c50d870c-10db-4f47-9809-de6b8c368c84" containerName="revoke-keys" Oct 30 01:41:56.540046 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.539935 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50d870c-10db-4f47-9809-de6b8c368c84" containerName="revoke-keys" Oct 30 01:41:56.540109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.540093 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50d870c-10db-4f47-9809-de6b8c368c84" containerName="revoke-keys" Oct 30 01:41:56.542809 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.542787 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.555646 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.555624 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:56.555782 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.555761 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:56.565609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.565585 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:56.565609 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.565589 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:41:56.567738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.567716 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff"] Oct 30 01:41:56.639943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.639910 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.640101 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.639984 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.640101 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.640047 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bm8\" (UniqueName: \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.741093 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.741060 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bm8\" (UniqueName: \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.741233 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.741132 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.741233 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.741177 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.741764 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.741736 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.743575 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.743549 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.759086 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.759062 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bm8\" (UniqueName: \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:56.855998 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:56.855916 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:41:57.160220 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:57.160193 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff"] Oct 30 01:41:57.162627 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:41:57.162592 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d4a350_2b3a_4bd3_b8b3_f900814084e8.slice/crio-2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f WatchSource:0}: Error finding container 2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f: Status 404 returned error can't find the container with id 2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f Oct 30 01:41:57.884647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:57.884616 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" event={"ID":"49d4a350-2b3a-4bd3-b8b3-f900814084e8","Type":"ContainerStarted","Data":"61c3a7391bd75781c27045d4d02cca614495a20052c3f98d083040ac7e57e1d4"} Oct 30 01:41:57.885115 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:57.884665 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" event={"ID":"49d4a350-2b3a-4bd3-b8b3-f900814084e8","Type":"ContainerStarted","Data":"2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f"} Oct 30 01:41:57.918412 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:57.918360 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" podStartSLOduration=1.9183465229999999 podStartE2EDuration="1.918346523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:57.917751846 +0000 UTC m=+3508.760312439" watchObservedRunningTime="2026-10-30 01:41:57.918346523 +0000 UTC m=+3508.760907166" Oct 30 01:41:58.891313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:58.891275 5548 generic.go:356] "Generic (PLEG): container finished" podID="49d4a350-2b3a-4bd3-b8b3-f900814084e8" containerID="61c3a7391bd75781c27045d4d02cca614495a20052c3f98d083040ac7e57e1d4" exitCode=0 Oct 30 01:41:58.891815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:41:58.891372 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" event={"ID":"49d4a350-2b3a-4bd3-b8b3-f900814084e8","Type":"ContainerDied","Data":"61c3a7391bd75781c27045d4d02cca614495a20052c3f98d083040ac7e57e1d4"} Oct 30 01:42:00.033015 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.032990 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:42:00.177034 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.176969 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\") pod \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " Oct 30 01:42:00.177175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.177051 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\") pod \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " Oct 30 01:42:00.177175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.177100 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\" (UniqueName: \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\") pod \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\" (UID: \"49d4a350-2b3a-4bd3-b8b3-f900814084e8\") " Oct 30 01:42:00.177393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.177371 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca" pod "49d4a350-2b3a-4bd3-b8b3-f900814084e8" (UID: "49d4a350-2b3a-4bd3-b8b3-f900814084e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:00.179354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.179331 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca" pod "49d4a350-2b3a-4bd3-b8b3-f900814084e8" (UID: "49d4a350-2b3a-4bd3-b8b3-f900814084e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:00.179354 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.179347 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8" pod "49d4a350-2b3a-4bd3-b8b3-f900814084e8" (UID: "49d4a350-2b3a-4bd3-b8b3-f900814084e8"). InnerVolumeSpecName "kube-api-access-z5bm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:00.278091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.278061 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:00.278091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.278087 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bm8\" (UniqueName: \"kubernetes.io/projected/49d4a350-2b3a-4bd3-b8b3-f900814084e8-kube-api-access-z5bm8\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:00.278091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.278097 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49d4a350-2b3a-4bd3-b8b3-f900814084e8-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:00.902760 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.902728 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" Oct 30 01:42:00.902974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.902727 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff" event={"ID":"49d4a350-2b3a-4bd3-b8b3-f900814084e8","Type":"ContainerDied","Data":"2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f"} Oct 30 01:42:00.902974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:00.902887 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2138bbb337d7208de84b1e9cc27f891a5c6bc91850bfa9d985e21e5c02c98e8f" Oct 30 01:42:01.331505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:01.331429 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:42:09.863427 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:09.863389 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff"] Oct 30 01:42:09.871384 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:09.871353 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b63f8a-78fff"] Oct 30 01:42:11.263102 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:11.263068 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d4a350-2b3a-4bd3-b8b3-f900814084e8" path="/var/lib/kubelet/pods/49d4a350-2b3a-4bd3-b8b3-f900814084e8/volumes" Oct 30 01:42:16.351756 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.351686 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" podUID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" containerName="main" containerID="cri-o://a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6" gracePeriod=45 Oct 30 01:42:16.613588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.613529 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:42:16.639426 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639399 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639444 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639474 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639566 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639514 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639565 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\" (UniqueName: \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639651 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\") pod \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\" (UID: \"7ea8d42d-87c2-4331-a8ef-c6297bc73571\") " Oct 30 01:42:16.639762 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639694 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.639974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639818 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.639974 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.639934 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.640270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.640190 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.640353 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.640344 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.640403 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.640360 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.641934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.641908 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "kube-api-access-wt8kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:16.641934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.641917 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:16.642091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.642003 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm" pod "7ea8d42d-87c2-4331-a8ef-c6297bc73571" (UID: "7ea8d42d-87c2-4331-a8ef-c6297bc73571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.741245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.741210 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ea8d42d-87c2-4331-a8ef-c6297bc73571-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.741245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.741239 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ea8d42d-87c2-4331-a8ef-c6297bc73571-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.741245 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.741251 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8kc\" (UniqueName: \"kubernetes.io/projected/7ea8d42d-87c2-4331-a8ef-c6297bc73571-kube-api-access-wt8kc\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.988196 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.988162 5548 generic.go:356] "Generic (PLEG): container finished" podID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" containerID="a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6" exitCode=0 Oct 30 01:42:16.988386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.988205 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" Oct 30 01:42:16.988386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.988210 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" event={"ID":"7ea8d42d-87c2-4331-a8ef-c6297bc73571","Type":"ContainerDied","Data":"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6"} Oct 30 01:42:16.988386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.988247 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7" event={"ID":"7ea8d42d-87c2-4331-a8ef-c6297bc73571","Type":"ContainerDied","Data":"c1e458e0c54cf166bcc1a0e5b8d7f20823a0563efef617f1b8529fe037583a42"} Oct 30 01:42:16.988386 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.988264 5548 scope.go:122] "RemoveContainer" containerID="a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6" Oct 30 01:42:16.997906 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.997885 5548 scope.go:122] "RemoveContainer" containerID="a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6" Oct 30 01:42:16.998191 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:42:16.998172 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6\": container with ID starting with a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6 not found: ID does not exist" containerID="a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6" Oct 30 01:42:16.998270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:16.998204 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6"} err="failed to get container status \"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6\": rpc error: code = NotFound desc = could not find container \"a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6\": container with ID starting with a8b0d30a9aef2fd579336e0084a6940ebe1654f2eac7b492793102073d035df6 not found: ID does not exist" Oct 30 01:42:17.034405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:17.034377 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:42:17.037382 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:17.037359 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b63f8a/simulator-single-main-b63f8a-kserve-5cbb546ff6-7h2j7"] Oct 30 01:42:17.263729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:17.263644 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" path="/var/lib/kubelet/pods/7ea8d42d-87c2-4331-a8ef-c6297bc73571/volumes" Oct 30 01:42:20.299114 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.299055 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:42:20.299626 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.299605 5548 cpu_manager.go:414] "containerMap: removing container" podUID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" containerName="main" Oct 30 01:42:20.299704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.299628 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" containerName="main" Oct 30 01:42:20.299704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.299660 5548 cpu_manager.go:414] "containerMap: removing container" podUID="49d4a350-2b3a-4bd3-b8b3-f900814084e8" containerName="revoke-keys" Oct 30 01:42:20.299704 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.299665 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d4a350-2b3a-4bd3-b8b3-f900814084e8" containerName="revoke-keys" Oct 30 01:42:20.303140 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.303112 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d4a350-2b3a-4bd3-b8b3-f900814084e8" containerName="revoke-keys" Oct 30 01:42:20.303347 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.303335 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea8d42d-87c2-4331-a8ef-c6297bc73571" containerName="main" Oct 30 01:42:20.312132 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.312101 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.322424 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.322400 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:42:20.329895 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.329849 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:42:20.334627 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.334602 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-546f251f-kserve-self-signed-certs\"" Oct 30 01:42:20.334772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.334622 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:42:20.334772 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.334710 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4jj\"" Oct 30 01:42:20.373297 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373269 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.373459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373305 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.373459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373381 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.373541 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373493 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hqj\" (UniqueName: \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.373578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373546 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.373618 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.373594 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474393 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474357 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hqj\" (UniqueName: \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474413 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474430 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474453 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474474 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.474594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474498 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.475053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474824 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.475053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474967 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.475053 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.474999 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.476939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.476917 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.477116 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.477100 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.492967 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.492936 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hqj\" (UniqueName: \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\") pod \"e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.626916 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.626812 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:20.923903 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:20.923880 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:42:20.925778 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:42:20.925749 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a1941b8_6a08_4b32_add9_bdc9e9889308.slice/crio-3ceaa920e049734e6eb339e3b188b30c7c5b445ac91351a9bef9e4e5fb8cfac1 WatchSource:0}: Error finding container 3ceaa920e049734e6eb339e3b188b30c7c5b445ac91351a9bef9e4e5fb8cfac1: Status 404 returned error can't find the container with id 3ceaa920e049734e6eb339e3b188b30c7c5b445ac91351a9bef9e4e5fb8cfac1 Oct 30 01:42:21.015108 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:21.015072 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" event={"ID":"6a1941b8-6a08-4b32-add9-bdc9e9889308","Type":"ContainerStarted","Data":"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9"} Oct 30 01:42:21.015246 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:21.015123 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" event={"ID":"6a1941b8-6a08-4b32-add9-bdc9e9889308","Type":"ContainerStarted","Data":"3ceaa920e049734e6eb339e3b188b30c7c5b445ac91351a9bef9e4e5fb8cfac1"} Oct 30 01:42:21.049499 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:21.049451 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" podStartSLOduration=1.049436864 podStartE2EDuration="1.049436864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:21.04907146 +0000 UTC m=+3531.891632053" watchObservedRunningTime="2026-10-30 01:42:21.049436864 +0000 UTC m=+3531.891997456" Oct 30 01:42:21.627252 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:21.627219 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:25.462798 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:42:25.462759 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:42:30.627144 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:30.627101 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:30.640289 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:30.640256 5548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:31.096725 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:31.096695 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:42:55.427880 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:42:55.427827 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:42:58.219010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:58.218968 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:42:58.219648 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:42:58.219615 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:07.447080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:07.447025 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:43:10.787939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:10.786982 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-f5pvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:43:10.787939 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:10.787073 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-f5pvt" podUID="3a7c855c-7b8a-44a4-b19b-dfea17d5cda5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:13.222543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:13.221994 5548 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b88g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:43:13.222543 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:13.222076 5548 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b88g4" podUID="a262a9ba-188f-4898-9579-c12ae687dcb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:22.468747 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.468680 5548 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" podUID="6a1941b8-6a08-4b32-add9-bdc9e9889308" containerName="main" containerID="cri-o://991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9" gracePeriod=45 Oct 30 01:43:22.718516 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.718491 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:43:22.791787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791702 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\" (UniqueName: \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.791787 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791782 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.792060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791811 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.792060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791847 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.792060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791893 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.792060 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.791970 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\") pod \"6a1941b8-6a08-4b32-add9-bdc9e9889308\" (UID: \"6a1941b8-6a08-4b32-add9-bdc9e9889308\") " Oct 30 01:43:22.792299 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.792125 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:22.792359 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.792341 5548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-home\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:22.792416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.792359 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:22.792461 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.792428 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:22.794309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.794285 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:22.794309 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.794284 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:22.794500 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.794484 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj" pod "6a1941b8-6a08-4b32-add9-bdc9e9889308" (UID: "6a1941b8-6a08-4b32-add9-bdc9e9889308"). InnerVolumeSpecName "kube-api-access-m5hqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:22.892930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.892900 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hqj\" (UniqueName: \"kubernetes.io/projected/6a1941b8-6a08-4b32-add9-bdc9e9889308-kube-api-access-m5hqj\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:22.892930 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.892926 5548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-dshm\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:22.893107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.892940 5548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1941b8-6a08-4b32-add9-bdc9e9889308-tls-certs\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:22.893107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.892952 5548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-model-cache\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:22.893107 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:22.892987 5548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1941b8-6a08-4b32-add9-bdc9e9889308-tmp-dir\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.443118 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.443080 5548 generic.go:356] "Generic (PLEG): container finished" podID="6a1941b8-6a08-4b32-add9-bdc9e9889308" containerID="991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9" exitCode=0 Oct 30 01:43:23.443337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.443152 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" Oct 30 01:43:23.443337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.443150 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" event={"ID":"6a1941b8-6a08-4b32-add9-bdc9e9889308","Type":"ContainerDied","Data":"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9"} Oct 30 01:43:23.443337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.443278 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq" event={"ID":"6a1941b8-6a08-4b32-add9-bdc9e9889308","Type":"ContainerDied","Data":"3ceaa920e049734e6eb339e3b188b30c7c5b445ac91351a9bef9e4e5fb8cfac1"} Oct 30 01:43:23.443337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.443309 5548 scope.go:122] "RemoveContainer" containerID="991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9" Oct 30 01:43:23.452578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.452562 5548 scope.go:122] "RemoveContainer" containerID="991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9" Oct 30 01:43:23.452889 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:43:23.452848 5548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9\": container with ID starting with 991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9 not found: ID does not exist" containerID="991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9" Oct 30 01:43:23.452978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.452902 5548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9"} err="failed to get container status \"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9\": rpc error: code = NotFound desc = could not find container \"991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9\": container with ID starting with 991aec9c3952e10cd09fd7628bb1e158017d4a5a880549a71918e36d52c20fb9 not found: ID does not exist" Oct 30 01:43:23.488940 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.488906 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:43:23.498807 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:23.498778 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-546f251f-kserve-69b5d96b8f-jmjkq"] Oct 30 01:43:25.033799 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:25.033764 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j"] Oct 30 01:43:25.043589 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:25.043561 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f89ac609-sr46j"] Oct 30 01:43:25.262890 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:25.262822 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1941b8-6a08-4b32-add9-bdc9e9889308" path="/var/lib/kubelet/pods/6a1941b8-6a08-4b32-add9-bdc9e9889308/volumes" Oct 30 01:43:25.263337 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:25.263314 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f1fd05-19d8-4dd4-9ec2-1c2a66480711" path="/var/lib/kubelet/pods/a5f1fd05-19d8-4dd4-9ec2-1c2a66480711/volumes" Oct 30 01:43:25.427334 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:43:25.427303 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:43:31.004964 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:31.004931 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:43:31.016776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:43:31.016752 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tcd47_69fd86b2-89ac-4e07-8dd7-da1cf88785c3/ovn-acl-logging/0.log" Oct 30 01:43:55.426093 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:43:55.426056 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:44:12.878748 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.878710 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb"] Oct 30 01:44:12.879677 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.879653 5548 cpu_manager.go:414] "containerMap: removing container" podUID="6a1941b8-6a08-4b32-add9-bdc9e9889308" containerName="main" Oct 30 01:44:12.879790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.879678 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1941b8-6a08-4b32-add9-bdc9e9889308" containerName="main" Oct 30 01:44:12.879790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.879781 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1941b8-6a08-4b32-add9-bdc9e9889308" containerName="main" Oct 30 01:44:12.882806 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.882787 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:12.906036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.906016 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:12.906036 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.906018 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:12.906206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.906182 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:12.906206 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.906196 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:44:12.908485 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:12.908466 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb"] Oct 30 01:44:13.055128 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.055100 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.055326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.055160 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzzx\" (UniqueName: \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.055326 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.055278 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.155779 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.155737 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzzx\" (UniqueName: \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.156006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.155787 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.156006 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.155851 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.156467 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.156444 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.158513 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.158492 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.180472 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.180449 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzzx\" (UniqueName: \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\") pod \"maas-api-revoke-sub-f03629db-g9jtb\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.195138 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.195115 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:13.507291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.507188 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb"] Oct 30 01:44:13.511336 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:44:13.511303 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d2dfef_6c70_44f9_8dce_358396a116c1.slice/crio-ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf WatchSource:0}: Error finding container ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf: Status 404 returned error can't find the container with id ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf Oct 30 01:44:13.753936 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:13.753901 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" event={"ID":"e2d2dfef-6c70-44f9-8dce-358396a116c1","Type":"ContainerStarted","Data":"ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf"} Oct 30 01:44:14.761313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:14.761279 5548 generic.go:356] "Generic (PLEG): container finished" podID="e2d2dfef-6c70-44f9-8dce-358396a116c1" containerID="70cbb2e4dfb8cf3eb57dbc4b786502b6a86e3c1997263285ce6da0e5440cdbc1" exitCode=0 Oct 30 01:44:14.761700 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:14.761378 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" event={"ID":"e2d2dfef-6c70-44f9-8dce-358396a116c1","Type":"ContainerDied","Data":"70cbb2e4dfb8cf3eb57dbc4b786502b6a86e3c1997263285ce6da0e5440cdbc1"} Oct 30 01:44:15.904091 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:15.904067 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:16.086706 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.086629 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\") pod \"e2d2dfef-6c70-44f9-8dce-358396a116c1\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " Oct 30 01:44:16.086882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.086721 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\" (UniqueName: \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\") pod \"e2d2dfef-6c70-44f9-8dce-358396a116c1\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " Oct 30 01:44:16.086882 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.086766 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\") pod \"e2d2dfef-6c70-44f9-8dce-358396a116c1\" (UID: \"e2d2dfef-6c70-44f9-8dce-358396a116c1\") " Oct 30 01:44:16.087190 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.087167 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca" pod "e2d2dfef-6c70-44f9-8dce-358396a116c1" (UID: "e2d2dfef-6c70-44f9-8dce-358396a116c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:16.088875 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.088837 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca" pod "e2d2dfef-6c70-44f9-8dce-358396a116c1" (UID: "e2d2dfef-6c70-44f9-8dce-358396a116c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:16.089028 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.089008 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx" pod "e2d2dfef-6c70-44f9-8dce-358396a116c1" (UID: "e2d2dfef-6c70-44f9-8dce-358396a116c1"). InnerVolumeSpecName "kube-api-access-mqzzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:16.188031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.187999 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:16.188031 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.188028 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d2dfef-6c70-44f9-8dce-358396a116c1-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:16.188201 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.188041 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzzx\" (UniqueName: \"kubernetes.io/projected/e2d2dfef-6c70-44f9-8dce-358396a116c1-kube-api-access-mqzzx\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:16.772295 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.772262 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" Oct 30 01:44:16.772473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.772291 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-g9jtb" event={"ID":"e2d2dfef-6c70-44f9-8dce-358396a116c1","Type":"ContainerDied","Data":"ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf"} Oct 30 01:44:16.772473 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:16.772341 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad721267eae8ed71ac0f6e045bd787cd8952c365bc054436c91e0506e4d88bdf" Oct 30 01:44:25.440710 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:44:25.440672 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:44:26.563010 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.562972 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c"] Oct 30 01:44:26.563592 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.563571 5548 cpu_manager.go:414] "containerMap: removing container" podUID="e2d2dfef-6c70-44f9-8dce-358396a116c1" containerName="revoke-keys" Oct 30 01:44:26.563720 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.563594 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d2dfef-6c70-44f9-8dce-358396a116c1" containerName="revoke-keys" Oct 30 01:44:26.563790 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.563754 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d2dfef-6c70-44f9-8dce-358396a116c1" containerName="revoke-keys" Oct 30 01:44:26.567210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.567189 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.579899 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.579873 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:26.580109 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.580097 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:26.590210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.590187 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:26.590371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.590249 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:44:26.593111 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.593045 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c"] Oct 30 01:44:26.601738 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.601715 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldlk\" (UniqueName: \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.601892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.601770 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.601972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.601953 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.703413 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.703382 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.703606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.703435 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldlk\" (UniqueName: \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.703606 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.703485 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.704275 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.704186 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.705950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.705928 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.722634 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.722611 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldlk\" (UniqueName: \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\") pod \"maas-api-revoke-sub-0165d7d2-hwj7c\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:26.880647 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:26.880578 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:27.181329 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:27.181299 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c"] Oct 30 01:44:27.183334 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:44:27.183307 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21938ed0_1c72_4e9d_b2bd_94f5805f15eb.slice/crio-13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636 WatchSource:0}: Error finding container 13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636: Status 404 returned error can't find the container with id 13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636 Oct 30 01:44:27.851741 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:27.851705 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" event={"ID":"21938ed0-1c72-4e9d-b2bd-94f5805f15eb","Type":"ContainerStarted","Data":"028f32f219be63f5fcffc7329ead3415236c622e551f3a9baf7634799c688f4d"} Oct 30 01:44:27.852141 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:27.851755 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" event={"ID":"21938ed0-1c72-4e9d-b2bd-94f5805f15eb","Type":"ContainerStarted","Data":"13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636"} Oct 30 01:44:28.858519 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:28.858486 5548 generic.go:356] "Generic (PLEG): container finished" podID="21938ed0-1c72-4e9d-b2bd-94f5805f15eb" containerID="028f32f219be63f5fcffc7329ead3415236c622e551f3a9baf7634799c688f4d" exitCode=0 Oct 30 01:44:28.858968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:28.858579 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" event={"ID":"21938ed0-1c72-4e9d-b2bd-94f5805f15eb","Type":"ContainerDied","Data":"028f32f219be63f5fcffc7329ead3415236c622e551f3a9baf7634799c688f4d"} Oct 30 01:44:29.994559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:29.994532 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:30.137394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.137319 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\" (UniqueName: \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\") pod \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " Oct 30 01:44:30.137559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.137421 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\") pod \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " Oct 30 01:44:30.137559 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.137492 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\") pod \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\" (UID: \"21938ed0-1c72-4e9d-b2bd-94f5805f15eb\") " Oct 30 01:44:30.137776 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.137752 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca" pod "21938ed0-1c72-4e9d-b2bd-94f5805f15eb" (UID: "21938ed0-1c72-4e9d-b2bd-94f5805f15eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:30.139588 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.139567 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca" pod "21938ed0-1c72-4e9d-b2bd-94f5805f15eb" (UID: "21938ed0-1c72-4e9d-b2bd-94f5805f15eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:30.139663 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.139639 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk" pod "21938ed0-1c72-4e9d-b2bd-94f5805f15eb" (UID: "21938ed0-1c72-4e9d-b2bd-94f5805f15eb"). InnerVolumeSpecName "kube-api-access-wldlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.238471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.238440 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.238471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.238467 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldlk\" (UniqueName: \"kubernetes.io/projected/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-kube-api-access-wldlk\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.238636 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.238478 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21938ed0-1c72-4e9d-b2bd-94f5805f15eb-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.871549 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.871505 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" event={"ID":"21938ed0-1c72-4e9d-b2bd-94f5805f15eb","Type":"ContainerDied","Data":"13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636"} Oct 30 01:44:30.871892 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.871840 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13d403976127368a0aebdf91d4d3acd08033a28692d4745e0585547fcde6b636" Oct 30 01:44:30.875095 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:30.875067 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hwj7c" Oct 30 01:44:31.284387 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:31.284356 5548 scope.go:122] "RemoveContainer" containerID="1a5415d6de8f59ab5aa1e67549557e06b1726ca2555384746345408ef8993e35" Oct 30 01:44:37.470430 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.470390 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb"] Oct 30 01:44:37.471029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.471010 5548 cpu_manager.go:414] "containerMap: removing container" podUID="21938ed0-1c72-4e9d-b2bd-94f5805f15eb" containerName="revoke-keys" Oct 30 01:44:37.471131 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.471032 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21938ed0-1c72-4e9d-b2bd-94f5805f15eb" containerName="revoke-keys" Oct 30 01:44:37.471209 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.471179 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="21938ed0-1c72-4e9d-b2bd-94f5805f15eb" containerName="revoke-keys" Oct 30 01:44:37.474264 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.474244 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.496729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.496702 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:37.496992 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.496968 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:37.496992 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.496973 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:44:37.497213 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.497157 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:37.499182 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.499159 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb"] Oct 30 01:44:37.605767 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.605737 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.605968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.605801 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9sr\" (UniqueName: \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.605968 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.605921 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.707258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.707227 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9sr\" (UniqueName: \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.707258 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.707265 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.707471 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.707326 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.708024 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.708005 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.709775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.709754 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.725105 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.725038 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9sr\" (UniqueName: \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\") pod \"maas-api-revoke-sub-6b19338f-whgvb\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:37.788051 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:37.788008 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:38.087139 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:38.087112 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb"] Oct 30 01:44:38.088881 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:44:38.088837 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faaefb6_7013_4810_a2ed_f65a64e9794a.slice/crio-a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5 WatchSource:0}: Error finding container a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5: Status 404 returned error can't find the container with id a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5 Oct 30 01:44:38.915250 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:38.915215 5548 generic.go:356] "Generic (PLEG): container finished" podID="2faaefb6-7013-4810-a2ed-f65a64e9794a" containerID="ddcf9822c357b848390904d5a20497b39d273ea88579836e0fedf2b22ce09759" exitCode=0 Oct 30 01:44:38.915619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:38.915262 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" event={"ID":"2faaefb6-7013-4810-a2ed-f65a64e9794a","Type":"ContainerDied","Data":"ddcf9822c357b848390904d5a20497b39d273ea88579836e0fedf2b22ce09759"} Oct 30 01:44:38.915619 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:38.915341 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" event={"ID":"2faaefb6-7013-4810-a2ed-f65a64e9794a","Type":"ContainerStarted","Data":"a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5"} Oct 30 01:44:40.053642 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.053611 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:40.129817 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.129792 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\") pod \"2faaefb6-7013-4810-a2ed-f65a64e9794a\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " Oct 30 01:44:40.130025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.129905 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\") pod \"2faaefb6-7013-4810-a2ed-f65a64e9794a\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " Oct 30 01:44:40.130025 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.129935 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\" (UniqueName: \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\") pod \"2faaefb6-7013-4810-a2ed-f65a64e9794a\" (UID: \"2faaefb6-7013-4810-a2ed-f65a64e9794a\") " Oct 30 01:44:40.130165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.130125 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca" pod "2faaefb6-7013-4810-a2ed-f65a64e9794a" (UID: "2faaefb6-7013-4810-a2ed-f65a64e9794a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:40.132127 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.132099 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca" pod "2faaefb6-7013-4810-a2ed-f65a64e9794a" (UID: "2faaefb6-7013-4810-a2ed-f65a64e9794a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:40.132262 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.132179 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr" pod "2faaefb6-7013-4810-a2ed-f65a64e9794a" (UID: "2faaefb6-7013-4810-a2ed-f65a64e9794a"). InnerVolumeSpecName "kube-api-access-wb9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:40.230963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.230896 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-xks-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:40.230963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.230920 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb9sr\" (UniqueName: \"kubernetes.io/projected/2faaefb6-7013-4810-a2ed-f65a64e9794a-kube-api-access-wb9sr\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:40.230963 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.230931 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2faaefb6-7013-4810-a2ed-f65a64e9794a-maas-api-openshift-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:44:40.927175 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.927142 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" Oct 30 01:44:40.927389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.927178 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-whgvb" event={"ID":"2faaefb6-7013-4810-a2ed-f65a64e9794a","Type":"ContainerDied","Data":"a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5"} Oct 30 01:44:40.927389 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:40.927231 5548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12e9112a2b271685818f8b1a8260cea3bfec2d30009d1976dc94e56aa9721f5" Oct 30 01:44:55.427251 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:44:55.427210 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:44:59.170162 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:59.170133 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-96qsm_f8c64d48-a509-44d7-b54c-136d59eedcd9/manager/0.log" Oct 30 01:44:59.589737 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:44:59.589661 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s9xwl_4a80f3cc-cbc5-45fe-92f1-4b89ce732358/manager/0.log" Oct 30 01:45:00.092100 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.092070 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hqgk7_316952b3-df35-4305-8b20-0b5aa60957e7/server/0.log" Oct 30 01:45:00.154463 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.154428 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:00.154848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.154821 5548 cpu_manager.go:414] "containerMap: removing container" podUID="2faaefb6-7013-4810-a2ed-f65a64e9794a" containerName="revoke-keys" Oct 30 01:45:00.154848 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.154839 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2faaefb6-7013-4810-a2ed-f65a64e9794a" containerName="revoke-keys" Oct 30 01:45:00.155083 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.155002 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="2faaefb6-7013-4810-a2ed-f65a64e9794a" containerName="revoke-keys" Oct 30 01:45:00.159496 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.159476 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.174912 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.174885 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:45:00.184815 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.184793 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:45:00.185051 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.184827 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:45:00.187883 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.187841 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:00.225092 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.225060 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.225263 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.225177 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnx87\" (UniqueName: \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.270755 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.270718 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-bjbg9_626858bc-fea2-4656-9704-0684969f1490/manager/0.log" Oct 30 01:45:00.325972 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.325947 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnx87\" (UniqueName: \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.326149 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.326044 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.326641 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.326620 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.344017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.343962 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnx87\" (UniqueName: \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\") pod \"maas-api-key-cleanup-29888745-gjpd8\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.472680 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.472629 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:00.483311 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.483288 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-d777n_a02e4f48-ac1a-4cfe-b406-1cd83d454a7c/manager/0.log" Oct 30 01:45:00.682884 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.682849 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-487tj_cda57833-6b29-49b5-a0f8-7bf80508dd8a/copy-manifests/0.log" Oct 30 01:45:00.708673 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.708646 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-487tj_cda57833-6b29-49b5-a0f8-7bf80508dd8a/manager/0.log" Oct 30 01:45:00.774210 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:00.774187 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:00.775739 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:45:00.775714 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899e6326_d432_4215_901a_db0f1e2a6cb6.slice/crio-9e3c7005a8a8bebd42647af3da3ba328c9a170046f152844d882d504da2dcb31 WatchSource:0}: Error finding container 9e3c7005a8a8bebd42647af3da3ba328c9a170046f152844d882d504da2dcb31: Status 404 returned error can't find the container with id 9e3c7005a8a8bebd42647af3da3ba328c9a170046f152844d882d504da2dcb31 Oct 30 01:45:01.041841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:01.041807 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" event={"ID":"899e6326-d432-4215-901a-db0f1e2a6cb6","Type":"ContainerStarted","Data":"9e3c7005a8a8bebd42647af3da3ba328c9a170046f152844d882d504da2dcb31"} Oct 30 01:45:02.048485 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:02.048451 5548 generic.go:356] "Generic (PLEG): container finished" podID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerID="18c75256ab7316a522c6b65dfcc96d2f1f7408e1065573bc8cbc12103a4a661d" exitCode=60 Oct 30 01:45:02.048934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:02.048538 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" event={"ID":"899e6326-d432-4215-901a-db0f1e2a6cb6","Type":"ContainerDied","Data":"18c75256ab7316a522c6b65dfcc96d2f1f7408e1065573bc8cbc12103a4a661d"} Oct 30 01:45:02.048934 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:02.048732 5548 scope.go:122] "RemoveContainer" containerID="18c75256ab7316a522c6b65dfcc96d2f1f7408e1065573bc8cbc12103a4a661d" Oct 30 01:45:02.519050 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:02.519024 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29888745-gjpd8_899e6326-d432-4215-901a-db0f1e2a6cb6/cleanup/0.log" Oct 30 01:45:02.699770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:02.699735 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hwj7c_21938ed0-1c72-4e9d-b2bd-94f5805f15eb/revoke-keys/0.log" Oct 30 01:45:03.055698 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.055609 5548 generic.go:356] "Generic (PLEG): container finished" podID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerID="f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324" exitCode=60 Oct 30 01:45:03.056142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.055692 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" event={"ID":"899e6326-d432-4215-901a-db0f1e2a6cb6","Type":"ContainerDied","Data":"f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324"} Oct 30 01:45:03.056142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.055753 5548 scope.go:122] "RemoveContainer" containerID="18c75256ab7316a522c6b65dfcc96d2f1f7408e1065573bc8cbc12103a4a661d" Oct 30 01:45:03.056142 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.056025 5548 scope.go:122] "RemoveContainer" containerID="f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324" Oct 30 01:45:03.056287 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:45:03.056226 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-gjpd8_odh-ai-gateway-infra(899e6326-d432-4215-901a-db0f1e2a6cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" Oct 30 01:45:03.255654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.255623 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-54a40efb-2n2hs_c50d870c-10db-4f47-9809-de6b8c368c84/revoke-keys/0.log" Oct 30 01:45:03.434184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:03.434156 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-whgvb_2faaefb6-7013-4810-a2ed-f65a64e9794a/revoke-keys/0.log" Oct 30 01:45:04.062821 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:04.062790 5548 scope.go:122] "RemoveContainer" containerID="f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324" Oct 30 01:45:04.063330 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:45:04.063070 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-gjpd8_odh-ai-gateway-infra(899e6326-d432-4215-901a-db0f1e2a6cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" Oct 30 01:45:04.139729 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:04.139705 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-g9jtb_e2d2dfef-6c70-44f9-8dce-358396a116c1/revoke-keys/0.log" Oct 30 01:45:04.738770 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:04.738739 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29888745-gjpd8_899e6326-d432-4215-901a-db0f1e2a6cb6/cleanup/1.log" Oct 30 01:45:07.190971 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:07.190939 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-szlp5_3b77d01f-40c1-4afe-bbf0-ec867396d4d0/manager/0.log" Oct 30 01:45:07.950850 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:07.950818 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g_54ef06e3-eb02-4a98-ac7a-910a12c8462b/istio-proxy/0.log" Oct 30 01:45:08.698038 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:08.698008 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-7r8hn_9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9/istio-proxy/0.log" Oct 30 01:45:09.088184 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:09.088093 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-b88g4_a262a9ba-188f-4898-9579-c12ae687dcb6/istio-proxy/0.log" Oct 30 01:45:09.473826 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:09.473796 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-f5pvt_3a7c855c-7b8a-44a4-b19b-dfea17d5cda5/istio-proxy/0.log" Oct 30 01:45:14.257315 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:14.257286 5548 scope.go:122] "RemoveContainer" containerID="f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324" Oct 30 01:45:15.128915 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:15.128885 5548 generic.go:356] "Generic (PLEG): container finished" podID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerID="f2b96a5366f5bd4925912e9d22375c166ad47d872cc4acd491e16525e700e672" exitCode=60 Oct 30 01:45:15.129033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:15.128958 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" event={"ID":"899e6326-d432-4215-901a-db0f1e2a6cb6","Type":"ContainerDied","Data":"f2b96a5366f5bd4925912e9d22375c166ad47d872cc4acd491e16525e700e672"} Oct 30 01:45:15.129088 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:15.129032 5548 scope.go:122] "RemoveContainer" containerID="f0e28ee59e7013d8e56db85132e798b5bfb9cd146db1c55b2f13052ddb056324" Oct 30 01:45:15.129358 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:15.129338 5548 scope.go:122] "RemoveContainer" containerID="f2b96a5366f5bd4925912e9d22375c166ad47d872cc4acd491e16525e700e672" Oct 30 01:45:15.129624 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:45:15.129605 5548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-gjpd8_odh-ai-gateway-infra(899e6326-d432-4215-901a-db0f1e2a6cb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" Oct 30 01:45:16.173707 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.173676 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:16.311014 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.310992 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:16.374594 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.374563 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\") pod \"899e6326-d432-4215-901a-db0f1e2a6cb6\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " Oct 30 01:45:16.374811 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.374611 5548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\" (UniqueName: \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\") pod \"899e6326-d432-4215-901a-db0f1e2a6cb6\" (UID: \"899e6326-d432-4215-901a-db0f1e2a6cb6\") " Oct 30 01:45:16.375033 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.375005 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca" pod "899e6326-d432-4215-901a-db0f1e2a6cb6" (UID: "899e6326-d432-4215-901a-db0f1e2a6cb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:16.375137 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.375115 5548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899e6326-d432-4215-901a-db0f1e2a6cb6-maas-api-service-ca\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:45:16.377933 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.377904 5548 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87" pod "899e6326-d432-4215-901a-db0f1e2a6cb6" (UID: "899e6326-d432-4215-901a-db0f1e2a6cb6"). InnerVolumeSpecName "kube-api-access-gnx87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:16.476556 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:16.476475 5548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnx87\" (UniqueName: \"kubernetes.io/projected/899e6326-d432-4215-901a-db0f1e2a6cb6-kube-api-access-gnx87\") on node \"ip-10-0-143-7.ec2.internal\" DevicePath \"\"" Oct 30 01:45:17.142935 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.142903 5548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" Oct 30 01:45:17.143135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.142902 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8" event={"ID":"899e6326-d432-4215-901a-db0f1e2a6cb6","Type":"ContainerDied","Data":"9e3c7005a8a8bebd42647af3da3ba328c9a170046f152844d882d504da2dcb31"} Oct 30 01:45:17.143135 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.143038 5548 scope.go:122] "RemoveContainer" containerID="f2b96a5366f5bd4925912e9d22375c166ad47d872cc4acd491e16525e700e672" Oct 30 01:45:17.193713 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.193687 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:17.203943 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.203921 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-gjpd8"] Oct 30 01:45:17.263267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:17.263237 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" path="/var/lib/kubelet/pods/899e6326-d432-4215-901a-db0f1e2a6cb6/volumes" Oct 30 01:45:22.636978 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:22.636947 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-96qsm_f8c64d48-a509-44d7-b54c-136d59eedcd9/manager/0.log" Oct 30 01:45:23.057321 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:23.057293 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s9xwl_4a80f3cc-cbc5-45fe-92f1-4b89ce732358/manager/0.log" Oct 30 01:45:23.550392 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:23.550361 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hqgk7_316952b3-df35-4305-8b20-0b5aa60957e7/server/0.log" Oct 30 01:45:23.778371 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:23.778342 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-bjbg9_626858bc-fea2-4656-9704-0684969f1490/manager/0.log" Oct 30 01:45:23.978352 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:23.978323 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-d777n_a02e4f48-ac1a-4cfe-b406-1cd83d454a7c/manager/0.log" Oct 30 01:45:24.199736 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:24.199709 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-487tj_cda57833-6b29-49b5-a0f8-7bf80508dd8a/copy-manifests/0.log" Oct 30 01:45:24.229067 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:24.228960 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-487tj_cda57833-6b29-49b5-a0f8-7bf80508dd8a/manager/0.log" Oct 30 01:45:25.427779 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:45:25.427745 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:45:25.872818 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:25.872745 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hwj7c_21938ed0-1c72-4e9d-b2bd-94f5805f15eb/revoke-keys/0.log" Oct 30 01:45:26.400441 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:26.400409 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-54a40efb-2n2hs_c50d870c-10db-4f47-9809-de6b8c368c84/revoke-keys/0.log" Oct 30 01:45:26.572459 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:26.572429 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-whgvb_2faaefb6-7013-4810-a2ed-f65a64e9794a/revoke-keys/0.log" Oct 30 01:45:27.276498 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:27.276471 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-g9jtb_e2d2dfef-6c70-44f9-8dce-358396a116c1/revoke-keys/0.log" Oct 30 01:45:29.704256 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:29.704218 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-szlp5_3b77d01f-40c1-4afe-bbf0-ec867396d4d0/manager/0.log" Oct 30 01:45:30.253727 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:30.253697 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g_54ef06e3-eb02-4a98-ac7a-910a12c8462b/istio-proxy/0.log" Oct 30 01:45:31.025757 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:31.025707 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-7r8hn_9c19f6f0-32e0-4fe0-8e52-0f83a859e6d9/istio-proxy/0.log" Oct 30 01:45:31.305151 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:31.305072 5548 scope.go:122] "RemoveContainer" containerID="2f3981f4a4029dad59a8ea8a82caa026b859d402b9ada18d9870a2336069b10f" Oct 30 01:45:31.402523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:31.402499 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-b88g4_a262a9ba-188f-4898-9579-c12ae687dcb6/istio-proxy/0.log" Oct 30 01:45:31.761136 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:45:31.761105 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-f5pvt_3a7c855c-7b8a-44a4-b19b-dfea17d5cda5/istio-proxy/0.log" Oct 30 01:45:55.426516 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:45:55.426472 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:46:25.426531 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:46:25.426489 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:46:49.033796 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:49.033507 5548 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs"] Oct 30 01:46:49.043165 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:49.043141 5548 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a40efb-2n2hs"] Oct 30 01:46:49.262505 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:49.262476 5548 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50d870c-10db-4f47-9809-de6b8c368c84" path="/var/lib/kubelet/pods/c50d870c-10db-4f47-9809-de6b8c368c84/volumes" Oct 30 01:46:51.665601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.665564 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvzh4/must-gather-2bp4p"] Oct 30 01:46:51.666068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.665992 5548 cpu_manager.go:414] "containerMap: removing container" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666005 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666017 5548 cpu_manager.go:414] "containerMap: removing container" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666068 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666022 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666091 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666100 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666107 5548 memory_manager.go:365] "RemoveStaleState removing state" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666171 5548 cpu_manager.go:414] "containerMap: removing container" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.666216 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.666176 5548 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899e6326-d432-4215-901a-db0f1e2a6cb6" containerName="cleanup" Oct 30 01:46:51.669339 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.669322 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.684055 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.684035 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"openshift-service-ca.crt\"" Oct 30 01:46:51.693709 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.693690 5548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"kube-root-ca.crt\"" Oct 30 01:46:51.693942 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.693781 5548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvzh4\"/\"default-dockercfg-lc9qq\"" Oct 30 01:46:51.701173 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.701148 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/must-gather-2bp4p"] Oct 30 01:46:51.723527 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.723496 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a592b9be-6040-4d9b-b4fe-42f36f989309-must-gather-output\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.723659 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.723618 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wqw\" (UniqueName: \"kubernetes.io/projected/a592b9be-6040-4d9b-b4fe-42f36f989309-kube-api-access-r7wqw\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.825003 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.824970 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wqw\" (UniqueName: \"kubernetes.io/projected/a592b9be-6040-4d9b-b4fe-42f36f989309-kube-api-access-r7wqw\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.825203 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.825053 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a592b9be-6040-4d9b-b4fe-42f36f989309-must-gather-output\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.825405 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.825384 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a592b9be-6040-4d9b-b4fe-42f36f989309-must-gather-output\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.845267 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.845234 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wqw\" (UniqueName: \"kubernetes.io/projected/a592b9be-6040-4d9b-b4fe-42f36f989309-kube-api-access-r7wqw\") pod \"must-gather-2bp4p\" (UID: \"a592b9be-6040-4d9b-b4fe-42f36f989309\") " pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:51.984334 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:51.984244 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" Oct 30 01:46:52.282394 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:52.282367 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/must-gather-2bp4p"] Oct 30 01:46:52.284997 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:46:52.284973 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda592b9be_6040_4d9b_b4fe_42f36f989309.slice/crio-c5b3f66de9fdb264639259dbbdbb514b698fe6a194a88a44e8a4adf46a296901 WatchSource:0}: Error finding container c5b3f66de9fdb264639259dbbdbb514b698fe6a194a88a44e8a4adf46a296901: Status 404 returned error can't find the container with id c5b3f66de9fdb264639259dbbdbb514b698fe6a194a88a44e8a4adf46a296901 Oct 30 01:46:52.286847 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:52.286830 5548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:46:52.649080 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:52.649049 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" event={"ID":"a592b9be-6040-4d9b-b4fe-42f36f989309","Type":"ContainerStarted","Data":"c5b3f66de9fdb264639259dbbdbb514b698fe6a194a88a44e8a4adf46a296901"} Oct 30 01:46:53.657852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:53.657808 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" event={"ID":"a592b9be-6040-4d9b-b4fe-42f36f989309","Type":"ContainerStarted","Data":"dcc7ece6f21adf07fa4053eaf5351d614caf2595e2364a64fa07f2a092b72984"} Oct 30 01:46:53.657852 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:53.657879 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" event={"ID":"a592b9be-6040-4d9b-b4fe-42f36f989309","Type":"ContainerStarted","Data":"3c223a806737a2c7c451a720455d8eba17be5c2ed077fa1db5f42a0202268734"} Oct 30 01:46:53.695012 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:53.694952 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tvzh4/must-gather-2bp4p" podStartSLOduration=1.922085839 podStartE2EDuration="2.694935228s" totalImagesPullingTime="772.849389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:53.69231082 +0000 UTC m=+3804.534871412" watchObservedRunningTime="2026-10-30 01:46:53.694935228 +0000 UTC m=+3804.537495828" Oct 30 01:46:55.433605 ip-10-0-143-7 kubenswrapper[5548]: E1030 01:46:55.433570 5548 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7kjw6" containerName="extractor" Oct 30 01:46:55.827075 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:55.826993 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ncxvc_1ab30191-af9f-4578-ae26-c57232aa5111/global-pull-secret-syncer/0.log" Oct 30 01:46:56.939331 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:56.939283 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ts2gs_368c928a-a45a-4446-942b-39bd53f20a04/konnectivity-agent/0.log" Oct 30 01:46:57.257089 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:46:57.256995 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-7.ec2.internal_09bade25a7c8ebe0a0dcc7613db38300/haproxy/0.log" Oct 30 01:47:04.509550 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:04.509514 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-szlp5_3b77d01f-40c1-4afe-bbf0-ec867396d4d0/manager/0.log" Oct 30 01:47:05.575303 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.575261 5548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74"] Oct 30 01:47:05.582950 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.582921 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.599349 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.597105 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74"] Oct 30 01:47:05.619017 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.618951 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-sys\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.619367 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.619336 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmj6g\" (UniqueName: \"kubernetes.io/projected/4e7bb0d3-189f-4007-ab1b-602958f7bf72-kube-api-access-rmj6g\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.619504 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.619444 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-lib-modules\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.619570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.619521 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-podres\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.619623 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.619573 5548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-proc\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.720982 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.720943 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-lib-modules\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.720982 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.720991 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-podres\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721016 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-proc\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721047 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-sys\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721096 5548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmj6g\" (UniqueName: \"kubernetes.io/projected/4e7bb0d3-189f-4007-ab1b-602958f7bf72-kube-api-access-rmj6g\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721120 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-lib-modules\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721204 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-sys\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721241 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721207 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-proc\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.721585 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.721264 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e7bb0d3-189f-4007-ab1b-602958f7bf72-podres\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.739523 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.739489 5548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmj6g\" (UniqueName: \"kubernetes.io/projected/4e7bb0d3-189f-4007-ab1b-602958f7bf72-kube-api-access-rmj6g\") pod \"perf-node-gather-daemonset-xvp74\" (UID: \"4e7bb0d3-189f-4007-ab1b-602958f7bf72\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.835578 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.835509 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/alertmanager/0.log" Oct 30 01:47:05.901369 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.901335 5548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:05.955134 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:05.955053 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/config-reloader/0.log" Oct 30 01:47:06.056335 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.056302 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/kube-rbac-proxy-web/0.log" Oct 30 01:47:06.168654 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.168626 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/kube-rbac-proxy/0.log" Oct 30 01:47:06.215291 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.215262 5548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74"] Oct 30 01:47:06.217270 ip-10-0-143-7 kubenswrapper[5548]: W1030 01:47:06.217228 5548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e7bb0d3_189f_4007_ab1b_602958f7bf72.slice/crio-3dc8a4a5289160f9f37b30ad00f686b4209b86f7c54918c2c1dd2829a8c9d1af WatchSource:0}: Error finding container 3dc8a4a5289160f9f37b30ad00f686b4209b86f7c54918c2c1dd2829a8c9d1af: Status 404 returned error can't find the container with id 3dc8a4a5289160f9f37b30ad00f686b4209b86f7c54918c2c1dd2829a8c9d1af Oct 30 01:47:06.273193 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.273157 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/kube-rbac-proxy-metric/0.log" Oct 30 01:47:06.380194 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.380169 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/prom-label-proxy/0.log" Oct 30 01:47:06.485902 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.485807 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a9097c6-bbb4-4ccb-aeec-3922b227652c/init-config-reloader/0.log" Oct 30 01:47:06.747676 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.747602 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zmdjf_40834bbb-90ae-486f-bbcb-ff665c3f313e/kube-state-metrics/0.log" Oct 30 01:47:06.796601 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.796560 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" event={"ID":"4e7bb0d3-189f-4007-ab1b-602958f7bf72","Type":"ContainerStarted","Data":"36b74c9eaee2ce0391779a929959e037dbbb42f12936f17d50ff0c80646a69f0"} Oct 30 01:47:06.796775 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.796755 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:06.796873 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.796804 5548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" event={"ID":"4e7bb0d3-189f-4007-ab1b-602958f7bf72","Type":"ContainerStarted","Data":"3dc8a4a5289160f9f37b30ad00f686b4209b86f7c54918c2c1dd2829a8c9d1af"} Oct 30 01:47:06.830617 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.830556 5548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" podStartSLOduration=1.8305361169999999 podStartE2EDuration="1.830536117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:06.829273148 +0000 UTC m=+3817.671833742" watchObservedRunningTime="2026-10-30 01:47:06.830536117 +0000 UTC m=+3817.673096702" Oct 30 01:47:06.847645 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.847592 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zmdjf_40834bbb-90ae-486f-bbcb-ff665c3f313e/kube-rbac-proxy-main/0.log" Oct 30 01:47:06.951270 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:06.951224 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zmdjf_40834bbb-90ae-486f-bbcb-ff665c3f313e/kube-rbac-proxy-self/0.log" Oct 30 01:47:07.059808 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:07.059741 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f9c87f897-7zng6_ed7102eb-69ac-4caf-8039-18466d995097/metrics-server/0.log" Oct 30 01:47:07.621813 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:07.621791 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mh62_aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/node-exporter/0.log" Oct 30 01:47:07.729638 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:07.729606 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mh62_aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/kube-rbac-proxy/0.log" Oct 30 01:47:07.831985 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:07.831958 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mh62_aab413ab-1d2a-4e3d-bb80-b6a3e7a8fb8d/init-textfile/0.log" Oct 30 01:47:08.276546 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:08.276514 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4cqjm_a80b6429-b22b-4105-a70a-799faed22645/kube-rbac-proxy-main/0.log" Oct 30 01:47:08.381490 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:08.381456 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4cqjm_a80b6429-b22b-4105-a70a-799faed22645/kube-rbac-proxy-self/0.log" Oct 30 01:47:08.483841 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:08.483809 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4cqjm_a80b6429-b22b-4105-a70a-799faed22645/openshift-state-metrics/0.log" Oct 30 01:47:09.716029 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:09.715997 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f4ccddd89-25bvh_11063556-5eff-4a8e-9005-5fe769c80fe7/telemeter-client/0.log" Oct 30 01:47:09.826233 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:09.826209 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f4ccddd89-25bvh_11063556-5eff-4a8e-9005-5fe769c80fe7/reload/0.log" Oct 30 01:47:09.929669 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:09.929639 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f4ccddd89-25bvh_11063556-5eff-4a8e-9005-5fe769c80fe7/kube-rbac-proxy/0.log" Oct 30 01:47:10.046072 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.045991 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/thanos-query/0.log" Oct 30 01:47:10.148804 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.148777 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/kube-rbac-proxy-web/0.log" Oct 30 01:47:10.250121 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.250092 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/kube-rbac-proxy/0.log" Oct 30 01:47:10.351812 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.351744 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/prom-label-proxy/0.log" Oct 30 01:47:10.456313 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.456278 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/kube-rbac-proxy-rules/0.log" Oct 30 01:47:10.557928 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:10.557897 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbf9b4559-mh9jz_7eeb58a8-a0fe-48f2-86a7-0e347a585522/kube-rbac-proxy-metrics/0.log" Oct 30 01:47:12.813977 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:12.813948 5548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-xvp74" Oct 30 01:47:17.440285 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:17.439674 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8n2dz_58ed2301-70f6-497f-b717-be08bdb686c4/dns/0.log" Oct 30 01:47:17.544416 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:17.544388 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8n2dz_58ed2301-70f6-497f-b717-be08bdb686c4/kube-rbac-proxy/0.log" Oct 30 01:47:17.902991 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:17.902961 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g_54ef06e3-eb02-4a98-ac7a-910a12c8462b/istio-proxy/0.log" Oct 30 01:47:17.990898 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:17.990843 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6qb5x_6266e78f-7ca5-43bf-b3e6-6206fc45c4c9/dns-node-resolver/0.log" Oct 30 01:47:19.390570 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:19.390544 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kp9nh_e469533b-597a-4cf2-afd5-ad2fa146a978/node-ca/0.log" Oct 30 01:47:21.312603 ip-10-0-143-7 kubenswrapper[5548]: I1030 01:47:21.312573 5548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccqmw9g_54ef06e3-eb02-4a98-ac7a-910a12c8462b/istio-proxy/0.log"